Abstract
Zoonotic disease surveillance presents a substantial problem in the management of public health. Globally, zoonoses have the potential to spread and negatively impact population health economic growth, and security. This research was conducted to investigate the current data sources, analytical methods, and limitations for cluster detection and prediction with particular interest in emerging bioinformatics tools and resources to inform the development of zoonotic surveillance spatial decision support systems. We recruited 10 local health personnel to participate in a Delphi study. Participants agreed cluster detection is a priority, though mathematical modeling methods and bioinformatics resources are not commonly used toward this endeavor. However, participants indicated a desire to utilize preventative measures. We identified many limitations for identifying clusters including software availability, appropriateness, training, and usage of emerging genetic data. Future decision support system development should focus on state health personnel priorities and tasks to better utilize emerging developments and available data.
Keywords: Delphi study, public health informatics, decision-making, zoonoses
BACKGROUND AND SIGNIFICANCE
Zoonotic diseases represent the majority of emerging infectious diseases,1 and present challenges for preparedness for health agencies. Public health institutions have long recognized the connection between geography and health, by maintaining surveillance of current and potential locations of zoonotic disease clusters or outbreaks.2 Clusters and outbreaks, often defined as a sudden increase in the expected number of cases, or an unusual aggregation of cases grouped in a place and time, are important indicators of possible threats to overall public health. Large datasets are necessary to support surveillance tasks such as cluster detection, which often include disease morbidity and mortality, environmental variables, or human mobility. Improvements in online data availability have made it possible to better integrate many sources relevant to disease clustering or outbreaks, as well as emerging data sources such as genetic data. There has been a recent emphasis on utilizing bioinformatics approaches in disease surveillance by leveraging the growing amount of sequence data and near real-time genomic sequencing.3 However, health agencies sparsely use bioinformatics tools and resources to organize and evaluate genetic data for surveillance of zoonotic viruses. A small set of emerging Geographic Information Systems (GIS) or spatial decision support applications have begun to consider viral genetics in addition to traditional spatiotemporal data for questions relating to viral dispersion and biodiversity. However, these tools do not focus on the surveillance tasks critical to health departments such as identification of clusters or hotspots of high-risk locations.4 This prevents health agencies from understanding how changes in the genome of the virus impact disease risk and spread. The integration of viral genetics and geospatial statistics is necessary to fully understand the emerging spatial patterns of zoonotic viruses by considering not only traditional epidemiological data such as location and timing of reported cases but also the genetics of the virus that causes the disease.
To successfully integrate bioinformatics tools and resources into applications designed for surveillance, the end user must be considered. Earlier work to develop surveillance tools for the public health domain has indicated that user involvement in the development process for any new software is vital to successful implementation, and matching the stakeholders’ needs.5,6 However, little is known about the current analytical practices, software, and data use of state and local health practitioners regarding cluster detection and prediction. Zoonotic disease surveillance has been studied to identify challenges of integrating animal and human zoonotic disease data to assist in the development of biomedical informatics tools using the Delphi method.7 Here, we elaborate on this type of work to facilitate a greater understanding of viral zoonotic disease surveillance tasks in local health departments including the use of bioinformatics resources. We applied the Delphi method to engage target users to determine currents tools and resources used in viral zoonotic disease outbreak surveillance, utility, or limitations.8,9 The results will support future projects to integrate bioinformatics tools and resources into spatial decision support systems that support the needs of public and anima health practitioners.
MATERIALS AND METHODS
Study design and development
The first round of a Delphi study is a survey of open-ended questions to elicit individual opinions on the study topic,10 and are typically informed by a literature review. We conducted a review of the literature addressing recent software and analytical methods used by state health practitioners for surveillance, using terms such as “public health” with “surveillance survey” or “capacity” (Google scholar search term example: (((Public health) AND surveillance) AND zoonotic) OR capacity) and used this information to develop preliminary questions. Additionally, we reviewed previous surveys developed for the target population11,12 which explored similar topics and were used to inform question development.
Expert selection
Target participants included epidemiologists, veterinarians, and wildlife health specialists who regularly handle and analyze data for zoonotic surveillance for state agencies of public health, agriculture, and wildlife within the United States. We identified the participants through their respective professional organizations, an approach used in previous work.13 Previous literature14,15 indicates 5–20 members chosen from a target population are sufficient for consensus. We considered geographical spread and equal representation during the recruitment process and, thus, selected an equal number of participants from each department, and at most 2 participants from an individual state. We selected 60 experts for participation as acceptance rates for Delphi studies have varied considerably from 30% to 100%.9,16
Delphi administration
We administered the Delphi study online, a process often referred to as an e-Delphi study, using online polling applications developed to assist in this process.17,18 For the purposes of survey distribution and analysis, we chose an online tool called Delphi Decision Aid.19 Once IRB approval was obtained, we contacted potential participants via email for consent and participation in the Delphi study. We distributed the initial survey used in the Delphi sessions as a series of open-ended questions (Table 1). All questions included in the preliminary survey were open ended, and if at least 2 of the participants initially identified an item as important or in common use, those items were retained for further consideration for the successive round. The following rounds contained Likert-scale questions on a 5-point ranked scale, ranging from strong disagreement1 to strong agreement5 and participants could provide additional commentary.20,21
Table 1.
Round 1 summary of feedback for questions asked of participants
| Question | Common themes and answers | |
|---|---|---|
| 1 | What viral zoonotic diseases does your department most often encounter? | Avian influenza, Eastern equine encephalitis, Hantavirus, Influenza A, Rabies virus, West Nile virus. |
| 2 | Is the detection of spatial clusters of disease a priority for the department? | Detection of spatial clusters of disease is a priority, methods such as algorithms to detect aberration from reports, simple increased reporting, or visualization of spatial distribution are used. |
| 3 | Is the prediction of spatial clusters of disease a priority for the department? | The prediction of spatial clusters of disease would be valuable for the department. Outside experts were used for modeling. Not performed locally. |
| 4 | Is mapping clusters of viral zoonotic disease a common task? | Mapping of clusters of viral zoonotic disease is a common task, for some diseases. |
| 5 | What software and data limitations are currently impacting assessment? | Limitations are resources for training, funding, speed of detection, real-time data collection, classifying rare events, no routine geocoding. |
| 6 | Are bioinformatics techniques or resources often used? | Bioinformatics techniques are not often used to assess viral zoonotic disease outbreaks/clusters. Sample data are often sent of the national agencies. |
| 7 | Please indicate priority data sources for surveillance and cluster detection. | Syndromic surveillance, morbidity/mortality, strain type, genetic data, demographic data are priory data sources for assessing disease outbreaks. |
| 8 | Is assessing a wide variety of covariates a priority? | Assessing a wide variety of covariates is not a priority when analyzing zoonotic disease clusters, because those included are discretionary. |
| 9 | Please indicate what types of Statistical analysis software are often used to assess zoonotic disease outbreaks/clusters. Examples include: (A) SAS, (B) R, (C) SPSS, (D) Excel, (E) Other. | The most common software used to assess zoonotic disease clusters includes SAS, R, ArcGIS, and Excel. |
| 10 | Please indicate what types of software suites are often used to analyze zoonotic disease outbreaks/clusters. Examples include: (A) EpiInfo, (B) None, (C) Other. | Software used to analyze outbreaks, developed by the local or national health institutions commonly include: EpiInfo or None. |
| 11 | Please indicate what types of GIS or spatial analysis software are often used to analyze zoonotic disease outbreaks/clusters. Examples include: (A) ArcGIS, (B) None, (C) other. | The most commonly used GIS software used is ArcGIS. |
| 12 | Please indicate what types of bioinformatics resources are used (A) GenBank, (B) Sequence alignment tools, (C) Variant typing, (D) None, (E) Other. | Bioinformatics resources such as GenBank, sequence alignment and variant typing are infrequently used. |
| 13 | Please indicate what types of cluster prediction methods are often used to analyze zoonotic disease outbreaks/clusters. Examples include: (A) Regression, (B) None, (C) Other. | Cluster prediction methods are uncommonly used to analyze zoonotic disease outbreaks/clusters. |
| 14 | In what other ways are clusters of disease outbreaks tracked and analyzed using software? | Clusters of disease outbreaks are often mapped, though inconsistent. |
| 15 | What limitations do you perceive with the software or other tools you use to detect and analyze zoonotic disease clusters? | Little data, data sharing or integration, training on geospatial software, and poor visualization tools are common problems. |
| 16 | Please indicate common collaborative activities to support surveillance tasks. | Collaboration to assess zoonotic disease among agencies is common, including use of surveillance data and consulting peers. |
Data collection and analysis
We recorded responses and rankings to determine participant agreement on items that should be included in the second round based on open responses to the initial round. To ensure stability in answers between participants between successive rounds, we used the stability factor.22 We selected a cutoff of <15% change between the group mean. Once stability in individual responses was reached for 80% of the questions, we calculated consensus. We measured consensus for within question agreement between 2 rounds using weighted Cohen’s kappa. We measured agreement among participants using Kendall’s W coefficient of concordance via R version 3.5.123 using the IRR24 package.
RESULTS
Participants
Ten individuals from 6 different Department of Health and Human Services regions (DHHS) participated in the preliminary round of the Delphi study. Participants were roughly evenly distributed among the 3 departments (4 from the department of agriculture, 3 from public health departments, and 3 from wildlife departments). Two-thirds of the participants held a leadership position in their division, while the remaining participants held intermediate positions. In Figure 1, we show the geographic distribution of participants. For the subsequent rounds, there were 6 participants. One withdrawal was due to the participant leaving their current position, otherwise reasons for withdrawal were unspecified.
Figure 1.
Distribution of participants by Department of Health and Human Services (DHHS) regions.
Preliminary round
Participant responses for the preliminary round (Table 1) indicted a variety of viral zoonotic diseases commonly studied, of which 6 were frequently agreed upon among at least 2 participants. Preliminary research into common software packages used for statistical analysis and mapping purposes aligned with participants responses, as few unique systems were mentioned, including SEDRIC25 and QGIS.26 Overall, participants agreed the detection and prediction of zoonotic disease clusters is an important task, and several agreed that detection methods are often used though prediction is not. Half of the participants indicated bioinformatics tools were not used in their department, though a variety of uses were described. We introduced additional items to the original question set to explore this topic starting in round 2.
Second and third round stability and consensus
While there were additional comments, most participants elaborated on agreeable components and none indicated a reworking or additional question. To establish stability of participant input, we distributed a third round with the same question set. In Table 2, we summarize the descriptive and inferential statistics calculated to compare the results of rounds 2 and 3. We calculated stability via the approach of Scheibe and Skutsch,27 using a less than 15% change level in mean scores to attain equilibrium. We achieved stability for all but 3 questions, reaching over 80%. This indicated little could be gained through additional rounds so we terminated the Delphi process.
Table 2.
Stability and consensus between rounds 2 and 3
| Questions | Round 2 ranking mean ± SD | Round 3 ranking mean ± SD | Stability (mean % change) | Kw |
|---|---|---|---|---|
| Q1 Priority viral zoonotic disease clusters tracked include: Avian Influenza, Hanta virus, Rabies, West Nile Virus, Influenza A, and Eastern Equine virus. | 4.29 ± 0.88 | 4.33 ± 0.75 | 0.93% (+0.04) | 0.75 |
| Q2. Detection of spatial clusters of disease is a priority. | 4.43 ± 0.73 | 4.5 ± 0.76 | 1.58% (+0.07) | 1 |
| Q3. The prediction of spatial clusters of disease is a priority. | 3.43 ± 1.18 | 3.17 ± 1.34 | 7.58% (−0.26) | 0.59 |
| Q4. Cluster prediction methods are uncommonly used to analyze viral zoonotic disease outbreaks/clusters. | 4.57 ± 0.73 | 3.83 ± 1.48 | 16.19% (−0.74) | 0.77 |
| Q5. Mapping of clusters of viral zoonotic disease is a common task. | 3 ± 1.31 | 3 ± 1.58 | 0% | 0.84 |
| Q6. The primary limitations for assessing viral zoonotic disease are: resources for training, funding, speed of detection, and data collection. | 4.14 ± 0.99 | 4 ± 1.15 | 3.38% (−0.14) | 0.86 |
| Q7. Available software is inappropriate/limiting factor in the detection/prediction of viral zoonotic disease clusters. | 2.71 ± 1.61 | 3 ± 1 | 10.7% (−0.29) | 0.67* |
| Q8. Assessing a wide variety of covariates is not a priority when analyzing viral zoonotic disease clusters. | 3 ± 0.82 | 3.5 ± 0.96 | 16.6% (+0.5) | 0.73 |
| Q9. Common software programs used to assess viral zoonotic diseases clusters include SAS, R, ArcGIS, and Excel. | 3.83 ± 1.07 | 3.67 ± 1.37 | 4.18% (−0.16) | 0.95 |
| Q10. Common software used to analyze outbreaks/clusters, developed by health institutions include: EpiInfo or None. | 2.83 ± 0.37 | 3.83 ± 0.9 | 35.5% (+1) | 0.14* |
| Q11. The most commonly used GIS software used is ArcGIS. | 4.33 ± 0.75 | 4.5 ± 0.5 | 3.92% (+0.17) | 0.80 |
| Q12. Training in the use of bioinformatics tools and resources is uncommon. | 3.83± 0.9 | 4 ±0.58 | 4.44% (+0.17) | 0.86 |
| Q13. Bioinformatics techniques are not often used to assess viral zoonotic disease clusters. | 4 ± 1.15 | 4.33 ± 0.75 | 8.25% (+0.33) | 0.83 |
| Q14. Syndromic surveillance, morbidity/mortality, strain type, genetic data, and demographic data are priory data sources for assessing viral zoonotic disease clusters. | 3.67 ± 1.37 | 3.83 ± 1.46 | 4.36% (+0.16) | 0.93 |
| Q15. Bioinformatics resources such as GenBank, sequence alignment and variant typing are infrequently used to analyze viral zoonotic disease clusters. | 3.5 ± 0.96 | 3.33 ± 0.37 | 4.86% (−0.17) | 0.82 |
| Q16. Including genetic data would provide additional insight into detecting and predicting viral zoonotic disease clusters. | 3.5 ± 1.12 | 3.67 ± 1.25 | 4.86% (+0.17) | 0.86 |
| Q17. Little data, training on geospatial software, and poor visualization tools are common problems. | 3.83 ± 1.34 | 4.33 ± 0.75 | 13.05% (+0.5) | 0.68 |
| Q18. Collaboration to assess viral zoonotic disease among agencies is common, including use of surveillance data and consulting peers. | 3.83 ± 0.07 | 4 ± 1.15 | 4.86% (+0.17) | 0.68 |
Note: All values are significant with P-values less than .05 unless otherwise indicated with an asterisk. Bolded items are those (3) questions which failed the 15% mean change cutoff.
We assessed within question agreement using a weighted kappa. Those questions which did not reach stability between rounds tended to have lower agreement. All but one question achieved at least moderate agreement, with half falling into the near perfect agreement range. Kendall’s W was 0.56 (0.41–0.61 for moderate agreement28), indicating moderate overall agreement.
Participant feedback and comments
Overall there was moderate to strong agreement rankings and kappa values for topics concerning the need to prioritize the detection of zoonotic diseases. Participants continued to agree on important zoonoses and cluster detection as a priority. Conversely, prediction was not identified as a priority, though a need to move away from reactionary methods to prevention measures was voiced. Participants also agreed on several data sources, methods, and tool usage including ArcGIS, and statistical software for detecting aberrations in case data. To detect clusters, participants used abnormal increases in reported cases rather than a statistical method such as a spatial scan. No tools specifically designed for spatial cluster detection were identified, though EpiInfo and SEDRIC were identified as useful for investigating anomalous events. When asked whether available tools were inappropriate for cluster detection and prediction, participants stated resources were not accessible either because they felt they did not have access or did not have the necessary training to utilize available software appropriately. There was moderate agreement that bioinformatics tools, resources, and training are uncommon. However, when asked whether genetic data sources were a priority and would help provide additional insight into cluster detection and prediction, participants moderately agreed.
DISCUSSION
Overall, several pertinent findings support previous research on public health priorities and short comings. Participants responses aligned with previous research identifying commonly targeted zoonotic diseases,29 and analysis tools that can assist in cluster detection are not effectively used by public health personnel.30 With the development of online resources and increasingly geocoded data, new decision support tools have become more readily available on a broader scale. The feedback given here also supports previous work indicating tools such as ArcGIS and SaTScan can be cumbersome for users not trained in their use to select the appropriate method or parameters.30 Furthermore, these tools were not originally designed for the end users targeted here, and have high potential for erroneous interpretation of the outputs.31 These limitations are also compounded by public health and related systems that are chronically lacking in resources, particularly personnel trained in informatics.32 Recent epidemiological capacity reporting also indicates that while many states are at near capacity for monitoring heath status and investigations, capacity for evaluations and research is low.32 Additional work is needed to explore the financial situation contributing to such limitations in software acquisition, use and development.
Feedback obtained in this study also shows an increasing interest in bioinformatics tools and resources, while a growing body of literature indicates that indeed there is an appreciation and need to utilize the genetic data to inform zoonotic disease surveillance efforts.33 Gardy and Loman34 propose that coupling sequencing of genomic data and enhanced surveillance and response could better support outbreaks and prevention needs, while Heesterbeek et al.35 also describes the progress in disease prevention and control as increasingly interconnected through organized surveillance on multiple temporal and spatial scales in relation to the environmental and evolutionary dynamics of infectious disease in both humans and animals. Prior to deployment of new tools and resources to address current limitations, public health practitioners should be included in the design process. This includes understanding the differences in desired outcomes and outputs from various types of stakeholders involved in zoonotic disease control, which can range from veterinary partners, to state epidemiologist and policymakers. Here, we provide a means by which future work can build on understanding the needs of health personnel to develop appropriate tools.
We note several limitations of this study including the small subset of experts. A larger sample size would have provided a more thorough understanding of various priorities and needs on a geographic and departmental level. As such, tools and resources in common usage within the United States for zoonotic disease cluster detection and prediction methods may have been overlooked. Finally, items generated for the preliminary round were based on literature reviews and similar surveys distributed to the target population and may have overlooked pertinent topics.
CONTRIBUTORS
R.B. and M.S. have made substantial contributions to the conception and design of the study, in addition to interpretation and the editing process.
FUNDING
This research was supported by a fellowship from the National Library of Medicine of the National Institutes of Health under award number F31LM012176 to R.B. and R01LM012080 to M.S.. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Conflict of interest statement. None declared.
REFERENCES
- 1. Jones KE, Patel NG, Levy MA, et al. Global trends in emerging infectious diseases. Nature 2008; 4517181: 990.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Caprarelli G, Fletcher S.. A brief review of spatial analysis concepts and tools used for mapping, containment and risk modelling of infectious diseases and other illnesses. Parasitology 2014; 1415: 581–601. [DOI] [PubMed] [Google Scholar]
- 3. Gardy J, Loman N, Rambaut A.. Real-time digital pathogen surveillance—the time is now. Genome Biol 2015; 16: 155. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. MacDonald N, Parks D, Beiko R.. SeqMonitor: influenza analysis pipeline and visualization. PLoS Curr 2009; 1: RRN1040. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Rayfield CHT, Will C, Adilya A, Sarah K, Mark A.. Integrated Disease Investigations and Surveillance planning: a systems approach to strengthening national surveillance and detection of events of public health importance in support of the International Health Regulations. BMC Public Health 2010; 101: S6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Merkord C, Liu Y, Mihretie A, et al. Integrating malaria surveillance with climate data for outbreak detection and forecasting: the EPIDEMIA system. Malar J 2017; 161: 89. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Scotch M, Rabinowitz P, Brandt C.. State-level zoonotic disease surveillance in the United States. Zoonoses Public Health 2011; 588: 523–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Yost J, Dobbins M, Traynor R, DeCorby K, Workentine S, Greco L.. Tools to support evidence-informed public health decision making. BMC Public Health 2014; 141: 728.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Zhao ZG, Cheng JQ, Xu SL, Hou WL, Richardus JH.. A quality assessment index framework for public health services: a Delphi study. Public Health 2015; 1291: 43–51. [DOI] [PubMed] [Google Scholar]
- 10. Murry JW, Hammons J.. Delphi—a versatile methodology for conducting qualitative research. Rev High Educ 1995; 184: 423–36. [Google Scholar]
- 11. Hadler JI, Lampkins R, Lemmings J, et al. Assessment of epidemiology capacity in state health departments—United States, 2013. MMWR Morb Mortal Wkly Rep 2015; 6414: 394–8. [PMC free article] [PubMed] [Google Scholar]
- 12. Luo H, Sotnikov S, McLees A, Stokes S.. Factors driving the adoption of quality improvement initiatives in local health departments: results from the 2010 profile study. J Public Health Manag Pract 2015; 212: 176–85. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Allen H, Katz R.. Perceptions of reportable disease lists by state officials in the United States. World Med Health Policy 2013; 51: 4. [Google Scholar]
- 14. Clayton M. Delphi: a technique to harness expert opinion for critical decision‐making tasks in education. Educ Psychol 1997; 174: 373–86. [Google Scholar]
- 15. Delbecq AL, Van de Ven AH, Gustafson DH.. Group techniques for program planning: a guide to nominal group and Delphi processes Scott Foresman; 1975.
- 16. Weir CR, Hicken BL, Rappaport H, Nebeker JR.. Crossing the quality chasm: the role of information technology departments. Am J Med Qual 2006; 216: 382–93. [DOI] [PubMed] [Google Scholar]
- 17. Cam KM, McKnight PE, Doctor JN. The delphi method online: medical expert consensus via the internet. Proc AMIA Symp2002: 990.
- 18. Weise J, Fisher KR, Trollor J.. Utility of a modified online Delphi method to define workforce competencies: lessons from the intellectual disability mental health core competencies project. J Policy Pract Intellect Disabil 2016; 131: 15–22. [Google Scholar]
- 19. Armstrong JS. [Delphi Decision Aid] Main. 2018. http://armstrong.wharton.upenn.edu/delphi2/; Accessed May 20, 2018.
- 20. Rickards G, Magee C, Artino AR Jr.. You can’t fix by analysis what you've spoiled by design: developing survey instruments and collecting validity evidence. J Grad Med Educ 2012; 44: 407–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. McColl E, Jacoby A, Thomas L, Soutter J, Bamford C.. Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technol Assess 2002; 531: 256.. [DOI] [PubMed] [Google Scholar]
- 22. Dajani JS, Sincoff MZ, Talley WK.. Stability and agreement criteria for the termination of Delphi studies. Technol Forecast Soc Change 1979; 131: 83–90. [Google Scholar]
- 23. R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2018. [Google Scholar]
- 24. Gamer M, Lemon J, Puspendra-Singh I.. irr: Various Coefficients of Interrater Reliability and Agreement R package version 0.84; 2012. The R Foundation. Vienna, Austria.
- 25. SEDRIC: System for Enteric Disease Response, Investigation, and Coordination | Foodborne Outbreaks | Food Safety | CDC; 2018. https://www.cdc.gov/foodsafety/outbreaks/investigating-outbreaks/sedric.html; Accessed May 21, 2018.
- 26. QGIS Geographic Information System. Open Source Geospatial Foundation Project; 2018. https://www.qgis.org/; Accessed May 21, 2018.
- 27. Scheibe M, Skutsch MJ.. Experiments in Delphi methodology. The Delphi Method: Techniques and Application. Reading, MA: Addison-Wesley; 1975.
- 28. Anthony D. Understanding Advanced Statistics: A Guide for Nurses and Health Care Researchers Churchill Livingstone; 1999.
- 29. Scotch M, Baarson B, Beard R, Lauder R, Varman A, Halden RU.. Examining the differences in format and characteristics of zoonotic virus surveillance data on state agency websites. J Med Internet Res 2013; 154: e90.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Sherman RL, Henry KA, Tannenbaum SL, Feaster DJ, Kobetz E, Lee DJ.. Applying spatial analysis tools in public health: an example using SaTScan to detect geographic targets for colorectal cancer screening interventions. Prev Chronic Dis 2014; 11: E41.. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Joyce K. “ To me it’s just another tool to help understand the evidence”: public health decision-makers’ perceptions of the value of geographical information systems (GIS). Health Place 2009; 153: 801–10. [DOI] [PubMed] [Google Scholar]
- 32. The Council of State and Territorial Epidemiologists. 2017 Epidemiology Capacity Assessment Report. Atlanta, GA: The Council of State and Territorial Epidemiologists; 2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Hadfield J., Megill C, Bell SM, et al. Nextstrain: real-time tracking of pathogen evolution. Bioinformatics 2018; 34: 4121–3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Gardy JL, Loman NJ.. Towards a genomics-informed, real-time, global pathogen surveillance system. Nat Rev Genet 2017; 191: 9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Heesterbeek H, Anderson R, Andreasen V, et al. Modeling infectious disease dynamics in the complex landscape of global health. Science (New York, NY) 2015; 3476227: aaa4339. [DOI] [PMC free article] [PubMed] [Google Scholar]