Abstract
Background
Zoonotic diseases account for over 60% of all communicable diseases causing illness in humans and 75% of recently emerging infectious diseases. As limited resources are available for the control and prevention of zoonotic diseases, it is necessary to prioritize diseases in order to direct resources into those with the greatest needs. The selection of criteria for prioritization has traditionally been on the basis of expert opinion; however, details of the methods used to identify criteria from expert opinion often are not published and a full range of criteria may not be captured by expert opinion.
Methodology/Principal Findings
This study used six focus groups to identify criteria for the prioritization of zoonotic diseases in Canada. Focus groups included people from the public, animal health professionals and human health professionals. A total of 59 criteria were identified for prioritizing zoonotic diseases. Human-related criteria accounted for the highest proportion of criteria identified (55%), followed by animal-related criteria (26%) then pathogen/disease-related criteria (19%).
Similarities and differences were observed in the identification and scoring of criteria for disease prioritization between groups; the public groups were strongly influenced by the individual-level of disease burden, the responsibility of the scientific community in disease prioritization and the experiences of recent events while the professional groups were influenced by the societal- and population-level of disease burden and political and public pressure.
Conclusions/Significance
This was the first study to describe a mixed semi-quantitative and qualitative approach to deriving criteria for disease prioritization. This was also the first study to involve the opinion of the general public regarding disease prioritization. The number of criteria identified highlights the difficulty in prioritizing zoonotic diseases. The method presented in this paper has formulated a comprehensive list of criteria that can be used to inform future disease prioritization studies.
Introduction
Zoonotic diseases are defined by the World Health Organization (WHO) as those that are naturally transmitted between vertebrate animals and humans. Zoonotic diseases account for over 60% of all communicable diseases causing illness in humans and 75% of recently emerging infectious diseases [1], [2]; each disease posing a varying degree of threat to public health. As limited resources are available for research, surveillance, control and prevention of zoonotic diseases, it is necessary to prioritize diseases in order to direct resources into those with the greatest needs.
A number of studies have attempted to methodically prioritize communicable diseases and pathogens of national and international public health concern 3,4,5,6,7,8,9,10. More recently, studies have focused on the prioritization of zoonotic diseases and pathogens [11], [12], [13], [14]. Although methodological approaches differ, priority-setting exercises typically follow a series of steps, these include (i) selecting a group of diseases/pathogens for prioritization; (ii) identifying a list of appropriate and measurable criteria to assess diseases/pathogens; (iii) defining a range of levels for each criterion; (iv) determining the relative importance by means of a weight or score for each level within each criterion; (v) assigning weights and/or scores on the selected criteria and levels and aggregating to produce an overall score for each disease/pathogen; and (vi) ranking diseases/pathogens by their overall score to derive a recommended list for prioritization. A cut-off score may apply for the inclusion or exclusion of diseases/pathogens from the priority list.
In a review of prioritization studies conducted between 1997 and 2011 (Tables 1 and 2), 11 studies identified a list of criteria to assess diseases/pathogens (step (ii)). The number of criteria ranged from 5 to 12 and was primarily selected on the basis of expert opinion; however, details of the methods used to identify those criteria from expert opinion were not published.
Table 1. Summary of methods used in criteria identification in prioritization exercises conducted between 1997 and 2004.
| Country or region (Study year) | Diseases or pathogens of interest | Number of criteria identified | Methods used in criteria identification | Reference(s) |
| United Kingdom (1997) | 33 communicable diseases and 8 generic disease groups | 6 | Not published, expert opinion (n = ?) | [5] |
| “The questionnaire's validity and its appropriateness as a tool was assessed by experts in communicable diseases to ensure that it covered the main areas and criteria needed for the priority setting exercise.” | ||||
| United Kingdom (1999) | 58 pathogens or communicable diseases and 11 generic disease groups | 5 | Not published, expert opinion (n = ?) | [4] |
| “The five criteria used in the survey to assess importance are similar to criteria that have been used in other priority setting exercises.” | ||||
| Canada (2000) | 43 communicable diseases | 10 | Not published, expert opinion and consensus of the subcommittee (n = ?). | [8] |
| “The subcommittee established 10 criteria to measure the importance of each disease.” | ||||
| France (2000–2001) | 37 non-food borne zoonoses | 6 | Not published, expert opinion (n = 10) | [11] |
| “An expert group decided on a set of scientific criteria that would present objective arguments to decision makers.” [Translated from French] | ||||
| WHO Eastern Europe (7 countries) (2002) | 53 communicable diseases | 8 | Not published, expert opinion (n = 24) | [3] |
| “ A total of eight criteria for assessment of importance were selected …” [no mention of the criteria selection process but the study was conducted by a panel of experts] | ||||
| Germany (2004) | 85 pathogens | 12 | Not published, expert opinion (n = 11) | [9], [15] |
| “As the main purpose of our work is to guide surveillance and research activities in the field of infectious-disease control and epidemiology in Germany, it is not surprising that most of our categories relate strongly to public health in general and to epidemiology in particular.” |
Table 2. Summary of methods used in criteria identification in prioritization exercises conducted between 2005 and 2011.
| Country or region (Study year) | Diseases or pathogens of interest | Number of criteria identified | Methods used in criteria identification | Reference(s) |
| Canada (2005) | 48 communicable diseases | 10 | Not published, expert opinion and consensus of the subcommittee (n = ?) [8] | [7] |
| Continuation of the work conducted in 2000 [8] | ||||
| France (2005–2008) | 37 non-food borne zoonoses | 6 | Not published, expert opinion (n = 16) | [14] |
| Continuation of the work from the initial meeting in 2000–2001 [11]. | ||||
| Belgium (2008) | 51 food- and water-borne zoonotic pathogens | 5 | Not published, expert opinion (n = ?) | [12] |
| “The choice of the criteria was made from the viewpoint to have a well-balanced representation of public and animal health criteria and to add a criterion “food” to comply with the aim of the study. Socioeconomic aspects in relation to public and animal health were also taken into consideration.” | ||||
| The Netherlands (2010) | 86 emerging zoonotic pathogens | 7 | Not published, expert opinion (n = ?). | [13] |
| “We quantified the risk to public health of emerging zoonoses by applying seven criteria that covered the complete pathway from introduction to societal impact.” | ||||
| “The model for priority setting presented here is based on criteria reflecting the epidemiology and societal impact of zoonotic diseases. Risk perception by the general public is not included in this model, but may post additional challenges to policy makers. Further work to include risk perception as a second dimension in the priority model is recommended.” | ||||
| Germany (2011) | 127 pathogens | 10 | Not published, expert opinion (n = 11+72) | [10] |
| “The twelve criteria used during our previous prioritization process in 2004 [9], [15] were further modified according to the feedback received from a broad group of different experts [27]. The newly suggested criteria and their three-tiered definitions were then reviewed by internal and external experts.” |
As the success of a prioritization exercise is largely determined by having an appropriate list of criteria to assess diseases and pathogens, the methods for deriving the list of criteria should be transparent and scientifically driven. Although expert opinion was the preferred method in the selection of criteria for prioritization (Tables 1 and 2), many of these studies questioned whether all potential criteria were considered and whether the entire range of criteria can be captured impartially by expert opinion [4], [11], [12], [13], [15]. Expert groups acknowledge their priorities may not reflect the priorities concerning the general public or decision makers, particularly under social or political pressure [11], [12], [13]. It is therefore rational to include these stakeholders in developing a list of criteria for disease prioritization. Identifying an approach, that is transparent, reproducible and engages the collective opinion of content experts, decision makers and the general public, may lead to the selection of more appropriate criteria for a scientifically valid and robust prioritization exercise.
The first objective of this paper was to identify criteria for the prioritization of zoonotic diseases in Canada. Multiple focus groups comprising people from the public, animal health professionals and human health professionals were conducted to achieve this objective. The second objective was to explore the similarities and differences in responses between the public and the professional focus groups.
Methods
Focus groups were used to identify a list of criteria to prioritize zoonoses [16]. Six focus groups were conducted over a five-week period in February and March of 2010, each group comprising eight to ten individuals (Table 3). A total of 54 individuals participated in the focus groups. Written informed consent was obtained from each participating individual. The groups were selected to reflect a range of demographic and professional characteristics. The number of groups were selected with the goal of reaching theoretical saturation [17], [18].
Table 3. Demographic characteristics of focus group participants.
| Focus Group Number | Public or Professional | Number of individuals in the group | Males | Females | Average age (range) | Highest level of Education | Number of criteria identified as a group |
| 1 | Public | 9 | 6 | 3 | 38.3 (21 to 56) | High school (4), college diploma (2), Bachelors degree (2), Masters degree (1); all in unrelated fields* | 24 |
| 2 | Public | 9 | 5 | 4 | 41.4 (20 to 70) | High school (2), college diploma (3), Bachelors degree (4); all in unrelated fields* | 27 |
| 3 | Public | 10 | 4 | 6 | 40.2 (21 to 67) | High school (5), college diploma (1), Bachelors degree (3), Masters degree (1); all in unrelated fields* | 24 |
| 4 | Professional (animal health experts and decision-makers) | 10 | 8 | 2 | 47.7 (39 to 59) | PhD (7) and/or DVM/DVSc (7) | 33 |
| 5 | Professional (human health experts and decision-makers) | 8 | 5 | 3 | 47.1 (33 to 52) | Diploma (1), Bachelors degree (1), Masters (1), PhD (4) and/or MD (1) | 25 |
| 6 | Professional (mix of animal and human health experts and decision-makers) | 8 (four animal health and four human health professionals) | 3 | 5 | 42.7 (30 to 58) | Masters (1), PhD (4) and/or MD (1) and/or DVM/DVSc (3) | 31 |
| Total | - | 54 (total); 9 (mean) | 31 (57%) | 23 (43%) | 42.8 (mean) | - | 164 (total); 27 (mean) |
*Excluded fields include medical and veterinary sciences, epidemiology and public health sciences, infectious diseases research including laboratory work on infectious diseases, nurses, and dentists.
Three groups comprised of individuals from the general public. These individuals were screened and disqualified if they were employed in any of the following fields: medical science, veterinary science, epidemiology and public health sciences, infectious diseases research including laboratory work on infectious diseases, nurses, dentists and animal health technicians. Two recruitment agencies were enlisted to recruit individuals by telephone in the three public focus groups (one group was conducted in Guelph, Canada while the remaining two were conducted in Toronto, Canada). Recruitment involved the random selection of individuals from an in-house database with specifications to include a range of demographic characteristics (gender, age and educational background) in each group. The response rate during the screening process was not requested or available for these focus groups. A financial incentive (C$50) was given to public group participants. The remaining three groups comprised of individuals employed exclusively in these professional fields; these included infectious disease epidemiologists, academic and practicing physicians and veterinarians, human and animal health laboratory microbiologists, pathologists and technicians, public health practitioners and policymakers at the local, provincial and national level with at least five years of work experience. Individuals in the professional groups were recruited by one of the authors (VN) by email invitation. Recruitment involved targeting selected individuals representing the range of professions listed above in the local areas (Toronto and Guelph) rather than targeting all relevant healthcare professionals in the local region. The response rate was 86% (31 of 36 individuals approached), of these, 28 (78% of the total individuals approached) were available at the set times and dates of the focus groups. The professional group individuals were divided into one group of animal health professionals, one group of human health professionals and the last group comprising a mix of the two. No financial incentive was given to professional group participants.
A total of 28 people participated in the public focus groups and 26 professionals participated in the professional focus groups (Table 3). There were slightly more males than females across all groups (57% to 43%) and within public groups (54% to 46%) and professional groups (62% to 38%). The age range was 20 to 70 years in the public groups and 30 to 59 years in the professional groups. The professional groups encompassed a narrower age range because only individuals in active employment with at least 5 years of work experience in their profession were selected to participate. The public groups were more likely to be high school and college graduates or hold a Bachelor's degree in an unrelated field (these fields were medical and veterinary sciences, epidemiology and public health sciences, infectious diseases research including laboratory work on infectious diseases, nursing and dentistry) while the professional groups were more likely to hold a PhD in the related fields and/or a professional degree (DVM, DVSc or MD).
An experienced moderator conducted all six focus groups using a prepared script (available as Information S1) to ensure consistency between groups. The group discussions were audio taped and transcribed for validation of the group exercises. A nominal group technique [19] was used to structure the group discussion. Participants were presented with the research question - ‘What are important characteristics of zoonotic diseases that should be considered in disease prioritization?’ and were informed that the objective of the session was to formulate a list of criteria, in order of importance, that could be used to prioritize zoonotic diseases for their control and prevention in Canada. The exact mode of control and prevention was not specified but could include regulation, management, vaccination, laboratory diagnosis, research and surveillance. A zoonosis was defined as a disease that is naturally transmitted between humans and animals, including vector-borne and enteric diseases, as well as diseases of animal-origin but primarily circulating amongst humans (for example, SARS and H1N1).
The first half of the focus group session required participants to identify a list of criteria to prioritize zoonoses; participants were presented with a comparison of two zoonoses (H1N1 and H5N1) as a prompt for identifying a list of criteria. Group discussion at this stage was discouraged and participants were asked to identify a list of criteria without consultation from other group members. Participants were given 20 minutes for this step. Once completed, participants were invited to share one criterion from their list in a round-robin format until every criterion on each participant's list had been shared. To avoid duplication, participants were asked to only share one criterion that the group had not previously shared. A flip chart was used to document the group's combined list of criteria. On completion, participants were given the opportunity to explain why certain criteria were identified and a group discussion was encouraged to clarify and discuss the list. The group was then asked to review the list and to remove or merge criteria that appeared to overlap.
The second half of the focus group session was used to apply scores to the list of criteria identified by the group. Participants were asked to score on a scale of 1 (least important) to 9 (most important) each criterion identified. As with the formulation of the list of criteria, participants were asked to score each criterion on their own. Scores from each participant were tabulated and the mean score per criterion was used to create a ranked list of criteria; this list was presented back to the group for discussion. Participants were encouraged to discuss why some criteria ranked low while others ranked high. For the three public focus groups, a second round of scoring was undertaken giving participants the opportunity to re-rank each criterion in light of group discussion. Due to time constraint, the three professional groups only participated in one round of scoring.
The two authors (VN and JMS) independently merged and condensed the lists of criteria from each focus group into one overall list by combining similar criteria together; this process was informed by reviewing the transcripts from the focus group discussions. The decisive factors for combining similar criteria together were that they described the same characteristic (for example, “ability or potential to control the disease” and “efficacy of current control methods”) and/or that they overlapped in their definitions (for example, “case-fatality rate in humans” and “mortality caused by the disease in people” were merged into the criterion mortality characteristics in humans). Although case-fatality and mortality have distinct epidemiological definitions, both include the total number of deaths caused by the disease in their derivation, which would result in duplicating the characteristic ‘total number of deaths caused by the disease’ in the final list of criteria had both been included.
On completion, the two authors examined their condensed lists together and reached consensus on the number of unique criteria identified across the six focus groups. A score for each unique criterion was calculated by averaging the mean scores across duplicate criteria within each unique criterion. The authors also divided the unique criteria into nine themes for analysis by consensus. These themes were the burden of illness, disease epidemiology, control measures, socioeconomic impact, diagnosis, disease knowledge, public awareness and concerns, international considerations and local considerations. Although the burden of illness can be considered a theme within disease epidemiology, it encompasses the range of individual-level physical burden of the disease; these include mortality, severity, and the duration of illness. Disease epidemiology was considered to be aspects of the force of infection driving disease burden at the population-level; these include incidence, prevalence, transmission potential, speed of transmission within individuals, speed of transmission between individuals, immunogenicity, endemicity and high-risk groups.
Results
Six separate lists of criteria were obtained from the focus groups; each group identified between 24 and 33 criteria during the exercise, a total of 164 criteria were identified across all groups (Table 3). As expected, criteria duplication was observed across groups. A final list of 59 unique criteria for prioritizing zoonotic diseases elicited from six focus groups using the nominal group technique [19] are presented in Tables 4, 5 and 6. The criteria are presented by the nine themes that are further divided into human-related, animal-related or pathogen/disease-related categories. Criteria are presented in order of highest mean score by theme, then by human-, animal- or pathogen/disease-related categories. The theme with the most number of criteria identified was disease epidemiology (43%), followed by control measures (17%), socioeconomic impact (12%), disease burden (10%), and diagnosis (7%). Human-related criteria accounted for the highest proportion of criteria identified (55%), followed by animal-related criteria (26%) then pathogen and disease-related criteria (19%).
Table 4. Fifty-nine unique criteria identified collectively by six focus groups for prioritizing zoonotic diseases in Canada (Part 1).
| Criteria | Theme | Human, animal, pathogen or disease | Public Groups | Professional Groups | Mean Score∧ | Frequency# | |||||
| 1 | 2 | 3 | AH+ | HH+ | A/H+ | ||||||
| 1 | Mortality characteristics in humans | Burden of illness | Human | 2‡ | 1 | 1 | 1 | 8.6 | 4 | ||
| 2 | Severity of illness in humans | Burden of illness | Human | 1 | 1 | 1 | 1 | 1 | 1 | 7.6 | 6 |
| 3 | Duration of illness in humans | Burden of illness | Human | 2 | 6.0 | 1 | |||||
| 4 | Co-infection in humans | Burden of illness | Human | 1 | 5.1 | 1 | |||||
| 5 | Reproductive consequences in humans | Burden of illness | Human | 1 | 4.4 | 1 | |||||
| 6 | Severity of illness in animals | Burden of illness | Animal | 1 | 1 | 1 | 1 | 6.8 | 4 | ||
| 7 | Combination of disease risk and probability of infection | Disease epidemiology | Human | 1 | 8.6 | 1 | |||||
| 8 | Potential for animal-to-human transmission | Disease epidemiology | Human | 1 | 1 | 8.2 | 2 | ||||
| 9 | Speed of transmission between humans | Disease epidemiology | Human | 1 | 1 | 1 | 8.0 | 3 | |||
| 10 | Incidence or prevalence in humans | Disease epidemiology | Human | 1 | 1 | 7.5 | 2 | ||||
| 11 | Potential for human-to-human transmission | Disease epidemiology | Human | 1 | 1 | 1 | 1 | 7.9 | 4 | ||
| 12 | Risk of transmission in humans | Disease epidemiology | Human | 1 | 7.6 | 1 | |||||
| 13 | Speed of disease spread in individuals (incubation period) | Disease epidemiology | Human | 1 | 2 | 2 | 7.0 | 3 | |||
| 14 | High-risk groups in humans | Disease epidemiology | Human | 1 | 1 | 1 | 1 | 1 | 1 | 6.5 | 6 |
| 15 | Immunogenicity in humans | Disease epidemiology | Human | 1 | 1 | 1 | 5.0 | 3 | |||
| 16 | Risk of endemicity in humans | Disease epidemiology | Human | 1 | 5.0 | 1 | |||||
| 17 | Impact of climate change on vectors and animal hosts | Disease epidemiology | Animal | 2 | 7.0 | 1 | |||||
| 18 | Incidence or prevalence in animals | Disease epidemiology | Animal | 1 | 1 | 5.5 | 2 | ||||
| 19 | Specific types of animals involved | Disease epidemiology | Animal | 1 | 1 | 6.3 | 2 | ||||
| 20 | Size of the reservoir host (animal and environment) | Disease epidemiology | Animal | 1 | 2 | 2 | 6.2 | 3 | |||
| 21 | Potential for animal-to-animal transmission | Disease epidemiology | Animal | 1 | 2 | 6.1 | 2 | ||||
| 22 | Potential for human-to-animal transmission | Disease epidemiology | Animal | 1 | 1 | 1 | 5.5 | 3 | |||
| 23 | Risk of endemicity in animals | Disease epidemiology | Animal | 1 | 5.1 | 1 | |||||
| 24 | High-risk groups in animals | Disease epidemiology | Animal | 1 | 4.5 | 1 | |||||
AH = animal health professional group, HH = human health professional group and A/H = mixed animal and human health professional group.
Mean score on a scale of 1 being the least important to 9 being the most important.
The number of focus groups that identified the unique criterion; 1 indicates only one focus group listed the criterion while 6 indicates all focus groups listed the criterion.
The number of times the unique criterion was identified by each focus group; 2 or more indicates the criterion was listed more than once but expressed in different terms by the focus group.
Table 5. Fifty-nine unique criteria identified collectively by six focus groups for prioritizing zoonotic diseases in Canada (Part 2).
| Criteria | Theme | Human, animal, pathogen or disease | Public Groups | Professional Groups | Mean Score∧ | Frequency# | |||||
| 1 | 2 | 3 | AH+ | HH+ | A/H+ | ||||||
| 25 | Mode of transmission | Disease epidemiology | Pathogen/disease | 1‡ | 1 | 1 | 1 | 1 | 1 | 7.2 | 6 |
| 26 | Pathogenicity and virulence of the pathogen | Disease epidemiology | Pathogen/disease | 1 | 2 | 6.7 | 2 | ||||
| 27 | Range of pathogens | Disease epidemiology | Pathogen/disease | 1 | 6.8 | 1 | |||||
| 28 | Disease trend | Disease epidemiology | Pathogen/disease | 1 | 1 | 1 | 1 | 6.2 | 4 | ||
| 29 | Ability of the pathogen to mutate and adapt to change | Disease epidemiology | Pathogen/disease | 1 | 1 | 2 | 3 | 5.8 | 4 | ||
| 30 | Endemicity of the disease due to climate | Disease epidemiology | Pathogen/disease | 1 | 1 | 4.1 | 2 | ||||
| 31 | Seasonality of the disease | Disease epidemiology | Pathogen/disease | 1 | 1 | 1 | 3.5 | 3 | |||
| 32 | Treatment and prevention in humans | Control measures | Human | 2 | 4 | 2 | 2 | 2 | 1 | 7.2 | 6 |
| 33 | Potential to eradicate the disease in humans | Control measures | Human | 1 | 1 | 1 | 6.5 | 3 | |||
| 34 | Visual cues to avoid the disease in humans | Control measures | Human | 1 | 1 | 1 | 6.0 | 3 | |||
| 35 | Surveillance in humans | Control measures | Human | 1 | 5.9 | 1 | |||||
| 36 | Human cause versus a natural cause | Control measures | Human | 1 | 5.9 | 1 | |||||
| 37 | Disease in human beyond control measures | Control measures | Human | 1 | 1 | 5.8 | 2 | ||||
| 38 | Vaccine/antiviral manufacturing time | Control measures | Human | 1 | 5.8 | 1 | |||||
| 39 | Potential to eradicate the disease in animals | Control measures | Animal | 1 | 6.5 | 1 | |||||
| 40 | Treatment and prevention in animals | Control measures | Animal | 1 | 2 | 1 | 6.2 | 3 | |||
| 41 | Surveillance in animals | Control measures | Animal | 1 | 5.6 | 1 | |||||
| 42 | Risk of bioterrorism | Socioeconomic impact | Human | 1 | 1 | 6.7 | 2 | ||||
| 43 | Political impact of the disease in humans | Socioeconomic impact | Human | 1 | 6.4 | 1 | |||||
| 44 | Risk to the food and water supply | Socioeconomic impact | Human | 1 | 2 | 6.4 | 2 | ||||
| 45 | Socioeconomic burden of the disease in humans | Socioeconomic impact | Human | 1 | 2 | 2 | 2 | 1 | 1 | 5.6 | 6 |
| 46 | Psychological impact in humans | Socioeconomic impact | Human | 1 | 5.4 | 1 | |||||
| 47 | Socioeconomic burden of the disease at the individual/farm level | Socioeconomic impact | Animal | 1 | 1 | 5.8 | 2 | ||||
| 48 | Socioeconomic burden of the disease on the industry | Socioeconomic impact | Animal | 1 | 1 | 1 | 2 | 6.3 | 4 | ||
AH = animal health professional group, HH = human health professional group and A/H = mixed animal and human health professional group.
Mean score on a scale of 1 being the least important to 9 being the most important.
The number of focus groups that identified the unique criterion; 1 indicates only one focus group listed the criterion while 6 indicates all focus groups listed the criterion.
The number of times the unique criterion was identified by each focus group; 2 or more indicates the criterion was listed more than once but expressed in different terms by the focus group.
Table 6. Fifty-nine unique criteria identified collectively by six focus groups for prioritizing zoonotic diseases in Canada (Part 3).
| Criteria | Theme | Human, animal, pathogen or disease | Public Groups | Professional Groups | Mean Score∧ | Frequency# | |||||
| 1 | 2 | 3 | AH+ | HH+ | A/H+ | ||||||
| 49 | Ease of diagnosis and lead-time in diagnosis in humans | Diagnosis | Human | 1‡ | 1 | 1 | 6.0 | 3 | |||
| 50 | Availability of diagnostic tests in humans | Diagnosis | Human | 1 | 6.3 | 1 | |||||
| 51 | Test characteristics in humans (accuracy, multiple tests) | Diagnosis | Human | 1 | 3.9 | 1 | |||||
| 52 | Availability of diagnostic tests in animals | Diagnosis | Animal | 1 | 6.0 | 1 | |||||
| 53 | Scientific knowledge of the disease | Disease knowledge | Pathogen/disease | 1 | 1 | 1 | 1 | 6.3 | 4 | ||
| 54 | Public disruption | Public awareness and concerns | Human | 1 | 7.7 | 1 | |||||
| 55 | Public awareness | Public awareness and concerns | Human | 1 | 7.5 | 1 | |||||
| 56 | Public perception | Public awareness and concerns | Human | 1 | 1 | 1 | 5.6 | 3 | |||
| 57 | Geographic distribution of the disease | International considerations | Pathogen/disease | 1 | 1 | 1 | 1 | 4.6 | 4 | ||
| 58 | Geographic source of the disease | International considerations | Pathogen/disease | 1 | 5.1 | 1 | |||||
| 59 | Disease occurring in jurisdiction of interest | Local considerations | Pathogen/disease | 1 | 7.0 | 1 | |||||
| TOTAL | 24 | 27 | 24 | 33 | 25 | 31 | 6.2 | 141 | |||
| Mean Score ∧ by groups (range) | 6.2 (3.6 to 8.6) | 6.2 (3.3 to 8.7) | - | - | |||||||
AH = animal health professional group, HH = human health professional group and A/H = mixed animal and human health professional group.
Mean score on a scale of 1 being the least important to 9 being the most important.
The number of focus groups that identified the unique criterion; 1 indicates only one focus group listed the criterion while 6 indicates all focus groups listed the criterion.
The number of times the unique criterion was identified by each focus group; 2 or more indicates the criterion was listed more than once but expressed in different terms by the focus group.
The public groups identified fewer criteria (35) than the professional groups (46); of these, 22 were common, 13 were unique to the public groups and 24 were unique to the professional groups. Criteria relating to disease epidemiology, socioeconomic impact, diagnosis, public awareness and concerns and local considerations were identified more often in the professional groups (Tables 4, 5 and 6). Control measures, burden of illness, international considerations and disease knowledge were identified more often in the public groups. Human-related criteria accounted for the highest proportion of listed criteria in both the public and professional groups. The professional groups identified more animal-related criteria than the public groups. Due to the qualitative approach, the terms used to describe differences between groups, for example, more/less, higher/lower, are only numeric as no statistical comparisons were performed.
A mean score was calculated for each criterion; a score closer to 9 indicated the criterion was most important while a score closer to 1 indicated the criterion was least important. The mean score for all criteria across all groups was 6.2 ranging from 3.5 (seasonality of the disease) to 8.6 (mortality characteristics in humans and combination of disease risk and probability of infection) (Tables 4, 5 and 6). The mean score for the public groups was 6.2 ranging from 3.6 to 8.6. The mean score for the professional groups was also 6.2 ranging from 3.3 to 8.7.
The top ten criteria ranked by mean score by public and professional groups are presented in Table 7. Mortality characteristics in humans was the highest scored criteria in both the public and professional groups. Other shared criteria in the top ten between the two groups included mode of transmission and potential for human-to-human transmission. The bottom ten criteria ranked by mean score by public and professional groups are presented in Table 8. Seasonality of the disease was the lowest scored criteria in both the public and the professional groups. Other shared criteria in the bottom ten between the two groups included geographic distribution of the disease and immunogenicity in humans.
Table 7. Top 10 criteria rank-ordered by mean score by the public and professional focus groups.
| Rank | Focus group | Criteria | Themes | Human, animal or pathogen/disease | Mean Score |
| 1 | Public | Mortality characteristics in humans+ | Burden of illness | Human | 8.6 |
| 2 | Public | Speed of transmission between humans | Disease epidemiology | Human | 8.0 |
| 3 | Public | Treatment and prevention in humans | Control measures | Human | 7.5 |
| 4 | Public | Severity of illness in animals | Burden of illness | Animal | 7.5 |
| 5 | Public | Public awareness | Public concerns | Human | 7.5 |
| 6 | Public | Scientific knowledge of the disease | Disease knowledge | Pathogen/disease | 7.5 |
| 7 | Public | Mode of transmission+ | Disease epidemiology | Pathogen | 7.4 |
| 8 | Public | Potential to eradicate the disease in humans | Control measures | Human | 7.3 |
| 9 | Public | Speed of disease spread in individuals (incubation period) | Disease epidemiology | Human | 7.0 |
| 10 | Public | Potential for human-to-human transmission+ | Disease epidemiology | Human | 6.9 |
| 1 | Professional | Mortality characteristics in humans+ | Burden of illness | Human | 8.7 |
| 2 | Professional | Combination of disease risk and probability of infection | Disease epidemiology | Human | 8.6 |
| 3 | Professional | Severity of illness in humans | Burden of illness | Human | 8.4 |
| 4 | Professional | Potential for human-to-human transmission+ | Disease epidemiology | Human | 8.2 |
| 5 | Professional | Potential for animal-to-human transmission | Disease epidemiology | Human | 8.2 |
| 6 | Professional | Public disruption | Public concerns | Human | 7.7 |
| 7 | Professional | Risk of transmission in humans | Epidemiology | Human | 7.6 |
| 8 | Professional | Incidence or prevalence in humans | Disease epidemiology | Human | 7.5 |
| 9 | Professional | Socioeconomic burden of the disease in humans | Socioeconomic impact | Human | 7.1 |
| = 10 | Professional | Impact of climate change on vectors and animal hosts | Disease epidemiology | Animal | 7.0 |
| = 10 | Professional | Mode of transmission+ | Disease epidemiology | Pathogen | 7.0 |
Shared criterion identified as one of top ten criteria by both the public and the professional groups.
Table 8. Bottom 10 criteria rank-ordered by mean score by the public and professional focus groups.
| Rank | Focus group | Criteria | Themes | Human, animal or pathogen/disease | Mean Score |
| 35 | Public | Seasonality of the disease+ | Disease epidemiology | Disease | 3.6 |
| 34 | Public | Endemicity of the disease due to climate | Disease epidemiology | Disease | 4.1 |
| 33 | Public | Socioeconomic burden of the disease in humans | Socioeconomic impact | Human | 4.1 |
| 32 | Public | Reproductive consequences in humans | Burden of illness | Human | 4.4 |
| 31 | Public | Geographic distribution of the disease+ | International considerations | Pathogen/disease | 4.8 |
| 30 | Public | Disease in human beyond control measures | Control measures | Human | 4.8 |
| 29 | Public | Disease trend | Disease epidemiology | Disease | 5.0 |
| 28 | Public | Co-infection in humans | Burden of illness | Human | 5.1 |
| 27 | Public | Geographic source of the disease | International considerations | Pathogen/disease | 5.1 |
| 26 | Public | Immunogenicity in humans+ | Disease epidemiology | Human | 5.2 |
| 46 | Professional | Seasonality of the disease+ | Disease epidemiology | Disease | 3.3 |
| 45 | Professional | Test characteristics in humans (accuracy, multiple tests) | Diagnosis | Human | 3.9 |
| 44 | Professional | Geographic distribution of the disease+ | International considerations | Pathogen/disease | 4.0 |
| 43 | Professional | High-risk groups in animals | Disease epidemiology | Animal | 4.5 |
| 42 | Professional | Immunogenicity in humans+ | Disease epidemiology | Human | 4.6 |
| 41 | Professional | Ease of diagnosis and lead-time in diagnosis in humans | Diagnosis | Human | 4.9 |
| 40 | Professional | Risk of endemicity in humans | Disease epidemiology | Human | 5.0 |
| 39 | Professional | Scientific knowledge of the disease | Disease knowledge | Pathogen/disease | 5.1 |
| 38 | Professional | Potential to eradicate the disease in humans | Control measures | Human | 5.1 |
| 37 | Professional | Risk of endemicity in animals | Disease epidemiology | Animal | 5.1 |
Shared criterion identified as one of bottom ten criteria by both the public and the professional groups.
A list of criteria most frequently identified by all groups is presented in Table 9. Five criteria were identified by all six focus groups - severity of illness in humans, treatment and prevention in humans, high-risk groups in humans, socioeconomic burden of disease in humans and mode of transmission. For the public group, the criteria speed of disease spread in individuals (incubation period), speed of transmission between humans and visual cues to avoid the disease in humans were identified by all three public groups but not in any professional groups. Conversely, the criteria potential for human to animal transmission and public perception were identified by all three professional groups but not in any public groups.
Table 9. List of criteria most frequently identified by the public and professional focus groups.
| Focus group | Criteria | Themes | Human, animal or pathogen/disease | Frequency |
| Public | Severity of illness in humans+ | Burden of illness | Human | 3 |
| Public | Treatment and prevention in humans+ | Control measures | Human | 3 |
| Public | High-risk groups in humans+ | Disease epidemiology | Human | 3 |
| Public | Socioeconomic burden of the disease in humans+ | Socioeconomic impact | Human | 3 |
| Public | Mode of transmission+ | Disease epidemiology | Pathogen | 3 |
| Public | Mortality characteristics in humans | Burden of illness | Human | 3 |
| Public | Geographic distribution of the disease | International considerations | Pathogen/disease | 3 |
| Public | Speed of disease spread in individuals (incubation period)* | Disease epidemiology | Human | 3 |
| Public | Speed of transmission between humans* | Disease epidemiology | Human | 3 |
| Public | Visual cues to avoid the disease in humans* | Control measures | Human | 3 |
| Professional | Severity of illness in humans+ | Burden of illness | Human | 3 |
| Professional | Treatment and prevention in humans+ | Control measures | Human | 3 |
| Professional | High-risk groups in humans+ | Disease epidemiology | Human | 3 |
| Professional | Socioeconomic burden of the disease in humans+ | Socioeconomic impact | Human | 3 |
| Professional | Mode of transmission+ | Disease epidemiology | Pathogen | 3 |
| Professional | Severity of illness in animals | Burden of illness | Animal | 3 |
| Professional | Disease trend | Disease epidemiology | Disease | 3 |
| Professional | Potential for human-to-human transmission | Disease epidemiology | Human | 3 |
| Professional | Socioeconomic burden of the disease on the industry | Socioeconomic impact | Animal | 3 |
| Professional | Public perception* | Public concerns | Human | 3 |
| Professional | Potential for human-to-animal transmission* | Disease epidemiology | Animal | 3 |
Shared criterion identified by all six focus groups.
*Unique criterion identified only by either the public or professional groups.
Criteria are not listed in any particular order. All listed criteria were identified by all three public or professional focus groups.
Discussion
We present on a stakeholder-informed, mixed semi-quantitative and qualitative approach to identify a list of criteria that can be used to prioritize zoonotic diseases in Canada. A total of 59 unique criteria were identified in this exercise across six focus groups. Comparing the list of 59 unique criteria with the criteria used in previous prioritization studies (Tables 1 and 2), a much larger number of criteria were identified in this study than have been previously used, although many of these studies involved diseases exclusively in humans hence animal-related criteria would not be expected to be included in these studies. Further, previously published works focused on specific aspects of disease prioritization or a specialized group of diseases, for example prioritizing solely for surveillance [7], [8] or prioritizing emerging diseases only [13]. Finally, two or more criteria from previous studies could be combined into one of our unique criterion, for example, the severity of illness and duration of illness in humans were often combined into morbidity in humans while disease incidence could be combined with severity and mortality [3], [4], [5], [11], [12], [13], [14], thus reducing the number of criteria considered.
Similarities and differences were observed between the public and the professional focus groups. Amongst the similarities were the number of criteria identified within groups, the mean score across all identified criteria and the range in scores across all identified criteria (Tables 4, 5 and 6). The scores given to the most important criteria and least important criteria by groups were also near identical in score and range (Tables 7 and 8). The concordance in the mean score and range across the most salient and least salient criteria by groups indicate an overall agreement with the use of the scoring system and the nominal group technique for the rating of criteria importance.
Similarities were also observed in the overlap of three identified criteria each in the top ten and bottom ten lists suggesting some agreement between the public and professional groups in the criteria that can inform disease prioritization. The agreement on the high scoring of mortality characteristics in humans, mode of transmission and potential for human-to-human transmission suggest these criteria are of greatest importance for both public and professional stakeholders. This was substantiated by the focus group discussions in which the implications for human mortality and the transmissibility of the disease in humans dominated the discussion on what should be considered the most influential characteristics for disease prioritization. Conversely, low scoring criteria - seasonality of the disease, geographic distribution of the disease and immunogenicity in humans did not dominate in the focus group discussions, this was not because these criteria were unimportant (in fact, one could argue that all 59 unique criterion identified in this study were important given their identification in the focus groups), but rather, in comparison to all other identified competing criteria, these criteria were deemed less important by both public and professional stakeholders. The arguments for reduced importance varied and included that the cooler climate in Canada was unlikely to support disease persistence throughout the year (seasonality of the disease), that diseases do not respect political boundaries and with global trade and travel, a disease in one country can spread across the world within days (geographic distribution of the disease), and that immunity status can be modified by vaccination or is only relevant for diseases where reoccurrence is of high concern (immunogenicity in humans).
More differences were observed than similarities between the public and the professional groups. Although the number of criteria identified within each group was similar, the professional groups collectively identified more criteria with over half of these unique to the group (Tables 4, 5 and 6). Further, the professional groups described criteria definitions in more detail than the public group. For example, for the criterion socioeconomic burden of the disease in humans, the definition for this criterion in the professional groups encompassed a range of direct and indirect costs to the burden of illness including “the societal impact, the commerce impact, the psychological impact beyond the impact of the illness”, “cost of treating and managing people”, “impact on daily living” and “it's broader, indirect things, isolation, psychological impact, lost labour … it encompasses things that are more than just the direct cost of illness”. In comparison, the public group's definition of socioeconomic burden was primarily the “direct cost of treatment” (identified by two groups), “cost of prevention” and the “cost to control disease spread”, although one group expanded the definition further to include the “effect on the health care system including cost, hospital bed and logistics”. The larger number of unique criteria identified and the broader criteria definition in the professional groups was not surprising due to the educational background and professional experiences relating to the prioritization of diseases in this group. The differences in criterion definition therefore influenced its scoring with the professional groups scoring socioeconomic burden of the disease in humans high (Table 7); largely on the basis of the impact of this criterion at the societal- and population-level while the public group scored this criterion low (Table 8); primarily on the basis of the impact of this criterion at the individual-level.
Differences could also arise from emphasis placed on who was responsible for addressing the criterion. For example, the criterion scientific knowledge of the disease scored high in the public groups (Table 7) but low in the professional groups (Table 8). Although the reasoning given when identifying the criterion were similar between the public (“if it is a brand new disease and nobody knows anything about it, say in the 1980s and AIDS popped up and nobody knew anything about it, then it's a high priority for me”) and the professionals (“there is a need for research … how much is known generally, for example, if it is a brand new disease, like SARS was when it first came out, then I think it's going to be high priority”), the scoring differed between the public and professional groups because the public group placed more emphasis on the responsibility of scientists in addressing the scientific knowledge of the disease. Some of the comments from the public groups included “Scientists are the ones that are going to fix this problem if they came up with a cure”, “Because they [scientists] are the ones that are doing the research” and “If nothing is known, then it is a high priority for scientists in terms of finding funding for prioritization”. In comparison, the professional groups made comments relating only to the criterion rather than who was responsible for addressing the criterion, in fact, responsibility was assumed for this group - “if we know a lot versus if we don't know much, then maybe we need to prioritize it to know more” and “from a research point of view, how much do you need to know about a disease, so in other words, how much money needs to be spent on research”.
The focus groups were conducted less than one year after the initial outbreak of the pandemic influenza H1N1 virus in April 2009 [20] and the subsequent rapid spread across the globe [21], [22]. The awareness of the rapid spread of the H1N1 pandemic was evident in two criteria identified in the public groups – speed of disease spread in individuals (incubation period) and speed of transmission between humans, these were also criteria that were only identified by all three public groups and not in any professional groups. In reference to the speed of transmission, one participant noted ‘with speed of transmission I mean it originates from Central America and it's already in North America a week later’. Direct reference to H1N1 was also made for the criteria ability to mutate – ‘I was just thinking if the swine flu kind of mutated [then it would be a problem].’ The criteria how quickly can the vaccine or antiviral be manufactured, although not made in direct reference to H1N1, likely reflected on the vaccine shortages throughout Canada during the H1N1 pandemic [23]. It was evident that the criteria identified by the public groups were influenced by recent events while this was not as apparent for the professional groups.
While recent events may influence the public groups, the professional groups were strongly influenced by public perception. The criterion public perception was identified by all three professional groups but not in any public groups. Although public perception was a central theme for the professional group discussions, it was often argued that public perception should not be a factor in disease prioritization, thus a low score. However, the professional group acknowledged that disease prioritization in practice was often driven by public and political pressure. This is best summed up by one of the professional participants: “I think generally, politically, that is what happens [priority is given to diseases of media attention]. A disease may not be that important really in terms of morbidity and mortality, but if the public is panicking, then the government will do something”.
There are limitations relating to the method described in this paper, first and foremost, there may have been a selection bias relating to how individuals were chosen to participate in the study; the use of an in-house recruitment agency database to recruit individuals from the public and targeting selected professionals in the local region rather than randomly selecting individuals from the general public and targeting all relevant healthcare professionals across Canada. Our study therefore makes the assumption that the local public and professional groups selected to participate are in fact representative of the Canadian public and Canadian healthcare professionals. There may have also been a response bias relating to those who agreed to participate in the study. The effect of these methodological biases on the study results is unknown. Second, the professional groups undertook one round of scoring while the public groups were asked to re-score their criteria after a group discussion on the first round of scoring. While this may influence the final scores within groups, the concordance in the mean score and range between the public and the professional groups indicate an overall agreement with the use of the scoring system regardless of the number of rounds of scoring that was undertaken. Additionally, the professional groups were asked whether their scores would change after their group discussion and it was unanimously agreed upon that their scores would not change. Conversely, the public groups all agreed on the value of having a group discussion to come up with a final round of scoring, the latter of which did change from the initial round of scoring.
The advantages and disadvantages of using the nominal group technique in focus groups have already been described by other papers [16], [19]. The major strengths in the nominal group technique are the ability to clarify and minimize the differences of opinion between multiple participants and ensuring equal participation in all members. The main weaknesses are that a certain amount of agreement must already exist in participating members and the amount of time necessary to prepare and execute the technique in each focus group. The method described in this paper has been used to develop criteria for consideration in other health care studies [24], [25], [26].
We presented on the use of focus groups to derive a list of criteria for the prioritization of zoonotic diseases in Canada. This was the first study to describe a semi-quantitative and qualitative mixed approach to deriving criteria for disease prioritization. This study was also the first to collectively engage public and professional stakeholders in disease prioritization. Similarities and differences were observed in the identification and scoring of criteria for disease prioritization between groups; the public groups were strongly influenced by the individual-level of disease burden, the responsibility of the scientific community in disease prioritization and the experiences of recent events while the professional groups were influenced by the societal- and population-level of disease burden and political and public pressure.
The number of unique criteria identified in this study highlights the number of factors that need to be considered jointly, and thus, the difficulty in prioritizing zoonotic diseases. The study showed that relying on expert opinion alone may limit the range of suitable criteria considered for disease prioritization. The method presented in this paper has formulated a comprehensive list of criteria that can be used to inform other disease prioritization studies.
Supporting Information
Focus group script.
(DOCX)
Footnotes
Competing Interests: The authors have declared that no competing interests exist.
Funding: VN is supported by the Canadian Institute of Health Research/Public Health Agency of Canada, Applied Public Health Research Chair Program. JMS holds the Canadian Institute of Health Research/Public Health Agency of Canada, Applied Public Health Research Chair. The funding agency had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
References
- 1.Heymann DL. Control of Communicable Diseases Manual; In: Heymann DL, editor. Washington D.C: American Public Health Association; 2008. [Google Scholar]
- 2.Taylor LH, Latham SM, Woolhouse ME. Risk factors for human disease emergence. Phil Trans R Soc Lond B. 2001;356:983–989. doi: 10.1098/rstb.2001.0888. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.WHO. The Dubrovnik plege on surveillance and prioritization of infectious diseases. 2003. Report on a WHO meeting in Bucharest, Romania 21–23 November, 2003. Copenhagen, WHO Regional Office for Europe: World Health Organisation.
- 4.Horby P, Rushdy A, Graham C, O'Mahony M. PHLS overview of Communicable Diseases 1999. Communicable Disease and Public Health. 2001;4:8–17. [PubMed] [Google Scholar]
- 5.Rushdy A, O'Mahony M. PHLS overview of communicable diseases 1997: results of a priority setting exercise. Communicable Disease Report Supplement. 1998;8:S1–12. [PubMed] [Google Scholar]
- 6.Weinberg J, Grimaud O, Newton L. Establishing priorities for European collaboration in communicable disease surveillance. European Journal of Public Health. 1999;9:236–240. [Google Scholar]
- 7.Doherty J. Final Report and Recommendations from the National Notifiable Diseases Working Group. Canadian Communicable Disease Report. 2006;32:211–225. [PubMed] [Google Scholar]
- 8.Doherty J. Establishing priorities for national communicable disease surveillance. Canadian Journal of Infectious Diseases. 2000;11:21–24. doi: 10.1155/2000/134624. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Krause G, Alpers K, Benzler J, Bremer V, Claus H, et al. How can infectious diseases be prioritized in public health. EMBO Reports. European Molecular Biology Organization. 2008;9:S22–27. doi: 10.1038/embor.2008.76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Balabanova Y, Gilsdorf A, Buda S, Burger R, Eckmanns T, et al. Communicable Diseases Prioritized for Surveillance and Epidemiological Research: Results of a Standardized Prioritization Procedure in Germany, 2011. PLoS ONE. 2011;6 doi: 10.1371/journal.pone.0025691. doi: e25691. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Institut de Veille Sanitaire. Définition des priorités dans le domaine des zoonoses non alimentaires 2000–2002. Paris, France: Institut de Veille Sanitaire; 2002. [Google Scholar]
- 12.Cardoen S, van Huffel X, Berkvens D, Quoilin S, Ducoffre G, et al. Evidence-based semiquantitative methodology for prioritization of foodborne zoonoses. Foodborne pathogens and disease. 2009;6:1083–1096. doi: 10.1089/fpd.2009.0291. [DOI] [PubMed] [Google Scholar]
- 13.Havelarr AH, van Rosse F, Bucura C, Toetenel MA, Haagsma JA, et al. Prioritizing Emerging Zoonoses in The Netherlands. PLoS ONE. 2010;5 doi: 10.1371/journal.pone.0013965. doi: 10.1371/journal.pone.0013965. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Institut de Veille Sanitaire. Définition des priorités dans le domaine des zoonoses non alimentaires 2008–2009. Paris, France: Institut de Veille Sanitaire; 2010. [Google Scholar]
- 15.Krause G, Alpers K, Benzler J, Bremer V, Claus H, et al. Prioritisation of infectious diseases in public health - call for comments. Eurosurveillance. 2008;13:1–6. doi: 10.2807/ese.13.40.18996-en. [DOI] [PubMed] [Google Scholar]
- 16.Kitzinger J. Introducing focus groups. British Medical Journal. 1995;311:299–302. doi: 10.1136/bmj.311.7000.299. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Lewis-Beck M, Bryman A, Liao T. The Sage encyclopedia of social science research methods, Volume 1. London, UK: Sage Publications; 2004. [Google Scholar]
- 18.Krueger R, Casey M. Focus groups: a practical guide for applied research. London, UK: Sage Publications; 2009. 240 [Google Scholar]
- 19.Jones J, Hunter D. Consensus methods for medical and health services research. British Medical Journal. 1995;311:376–380. doi: 10.1136/bmj.311.7001.376. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.CDC. Outbreak of swine-origin influenza A (H1N1) virus infection - Mexico, March–April 2009. Morbidity Mortality Weekly Report. 2009;58:467–470. [PubMed] [Google Scholar]
- 21.Darwood F, Jain S, Finelli L, Shaw M, Lindstrom S, et al. Emergence of a novel swine-origin influenza A (H1N1) virus in humans. The New England Journal of Medicine. 2009;360:2605–2615. doi: 10.1056/NEJMoa0903810. [DOI] [PubMed] [Google Scholar]
- 22.Butler D. Swine flu goes global. Nature. 2009;458:1082–1083. doi: 10.1038/4581082a. [DOI] [PubMed] [Google Scholar]
- 23.PHAC. Lessons Learned Review: Public Health Agency of Canada and Health Canada response to the 2009 H1N1 Pandemic. Ottawa: Public Health Agency of Canada and Health Canada; 2010. 106 [Google Scholar]
- 24.Sampietro-Colom L, Espallargues M, Rodríguez E, Comas M, Alonso J, et al. Wide social participation in prioritizing patients on waiting list for joint replacement: a conjoint analysis. Medical Decision Making. 2008;28:554–566. doi: 10.1177/0272989X08315235. [DOI] [PubMed] [Google Scholar]
- 25.Leach M, Segal L. Patient attributes warranting consideration in clinical practice guidelines, health workforce planning and policy. BMC Health Services Research. 2011;19:221. doi: 10.1186/1472-6963-11-221. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.McCance T, Telford L, Wilson J, Macleod O, Dowd A. Identifying key performance indicators for nursing and midwifery care using a consensus approach. Journal of Clinical Nursing. 2011 doi: 10.1111/j.1365-2702.2011.03820.x. doi: 10.1111/j.1365-2702.2011.03820.x. [Epub ahead of print] [DOI] [PubMed] [Google Scholar]
- 27.Gilsdorf A, Krause G. Prioritisation of infectious diseases in public health: feedback on the prioritisation methodology, 15 July 2008 to 15 January 2009. Euro Surveillance. 2011;16:pii:19861. [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Focus group script.
(DOCX)