Abstract
The Institute of Medicine has recommended that policy decisions about improvement of national public health systems be guided by sound scientific evidence. However, to date there is no national research agenda to help guide public health systems.
The Centers for Disease Control and Prevention was called upon to lead a collaborative consensus-based process to define key research questions and establish a framework to create opportunities to better coordinate, leverage, and identify public health resources, which are increasingly scarce. The public health systems research agenda that emerged from this process has 14 overarching priority research themes. This national agenda should stimulate and guide research to meet the urgent need to improve the nation’s public health systems.
Centers for Disease Control and Prevention, in collaboration with the Council on Linkages Between Academia and Public Health Practice and other public health systems partners, should develop a research agenda and estimate funding needed to build the evidence base that will guide policymaking for public health practice.
—Institute of Medicine1(p9)
In its 2002 report, the Institute of Medicine (IOM) recommended that an investment be made in developing a research agenda to guide policy decisions that shape public health practice.1 Similarly, the US Department of Health and Human Services, in Healthy People 2010: Understanding and Improving Health, recognized the need for a strong public health infrastructure that would provide the capacity to prepare for and respond to acute and chronic threats to the nation’s health. Healthy People 2010 developed a series of benchmark indicators for public health infrastructure that calls for a systematic approach to data gathering, analysis, and research.2
Since the publication of these 2 seminal reports, the pace of change in public health has been accelerating, owing in large measure to the environment resulting from the events of September 11, 2001; newly emerging threats (e.g., severe acute respiratory syndrome [SARS], the obesity epidemic); and dramatic shifts in funding for public health agencies.
These new challenges have placed additional strains on already stressed services, programs, and staff. If we are to build the capacity needed to meet the ever-expanding list of threats to the public’s health, it is essential that we first define public health systems, how they function, and what factors contribute to high performance. Mays et al. describe public health systems research as “a field of study that examines the organization, financing, and delivery of public health services within communities, and the impact of these services on public health.”3(p180) To date, no public health systems research agenda exists.
The relatively new field of public health systems research is related to, but distinct from, more well-established areas such as health services research. It has emerged within the last decade primarily because of the need to better understand how the level of development of national public health infrastructure and the multiplicity of organizational arrangements in public health affect health outcomes. There is still a need to fully investigate the diversity of public health agency structures and functions, how resources are used at the state and local levels, how public health performance can affect health status outcomes, and myriad other issues. Early research and practice-based efforts represent the foundation upon which future research can be conducted.
Initial work in public health systems research generally focused on identifying the roles, functions, and resources of public health agencies.4–11 Over time, research expanded beyond agency boundaries to explore partnerships within public health by investigating the concept of a public health system12,13 and by focusing on collaborations between public health and sectors such as medicine14 and managed care.15 These activities were catalyzed, in great part, by a series of reports issued by the IOM. The 1988 IOM report urged a stronger focus on exploring and building the governmental public health role, as well as the role of other partners involved in public health.16 The IOM reiterated this call to action in 199717 and again in 2002.1 Other contributions to public health literature have echoed this need.18–23 Concurrently with researchers, public health practitioners have begun to address these issues in the field; this is evidenced through efforts such as the National Public Health Performance Standards Program24 and the Turning Point initiative.25
A public health systems research agenda will be instrumental in catalyzing new research and practice-based initiatives and raising awareness about the importance of such endeavors. A consensus-based research agenda establishes a framework that not only creates opportunities to better coordinate, leverage, and identify resources and activities but also provides the scientific basis for policy decisions affecting our nation’s health. Previous experiences in setting research agenda priorities in behavioral health,26 clinical preventive services,27 community design and land-use choices,28 and public health workforce issues29 have been taken into account in the process of developing a public health systems research agenda.
DEVELOPING A RESEARCH AGENDA: THE PROCESS
The Centers for Disease Control and Prevention’s (CDC’s) Public Health Practice Program Office, Division of Public Health Systems Development and Research, outlined a 4-step process with the stated goal of developing a consensus-based public health systems research agenda and disseminating the research agenda to the public health community. The National Public Health Performance Standards Program defines public health systems as “the collection of public, private and voluntary entities, as well as individuals and informal associations, that contribute to the public’s health within a jurisdiction.”24
The process was guided by several basic principles established to ensure that priority research themes would be defined without undue personal, economic, or political influence. First, the end users of public health systems research, represented by associations of public health practitioners, should have a strong voice in establishing the research agenda. Second, wide participation of interested parties should be ensured. Third, participants would be encouraged to propose research themes that were based on scientific need and the priorities of practitioners, without regard for perceived political or financial feasibility. Finally, to ensure a consensus-based outcome, the research themes should be prioritized through a nominal group process, in which each individual is given the opportunity to vote on prioritizing each research theme following extensive group discussion.
Step 1: Conduct a Brainstorming Session to Solicit Input From CDC Researchers
Staff from the Division of Public Health Systems Development and Research held a brainstorming session in April 2003 with the objective of generating a draft of key elements of a research agenda. The outcome of the session was a draft document outlining 4 broad research categories, each encompassing a list of research topics. The research categories were (1) public health system description and improvement; (2) public health agency role in the system; (3) resources and capacity assessment for the health system; and (4) performance and health outcomes. Performance in this context was understood to be how well a public health system provided the 10 essential public health services,30 measured against model standards defined in the National Public Health Performance Standards.
Step 2: Engage National Partners to Refine the Draft of Broad Research Themes
To gather input and help refine the 4 broad research categories, the CDC conducted a conference call with representatives from 2 universities; Mathematica Policy Research, Inc; and national partner organizations representing public health practitioners (Table 1 ▶).
TABLE 1—
Participants in Setting Priorities for a National Public Health Systems Research Agenda, 2003
| No. Participants | |
| Centers for Disease Control and Prevention (CDC) | 15 |
| American Public Health Association (APHA) | 3 |
| Association of State and Territorial Health Officials (ASTHO) | 3 |
| National Association of County and City Health Officials (NACCHO) | 6 |
| National Association of Local Boards of Health (NALBOH) | 4 |
| Public Health Foundation (PHF) | 2 |
| National Network of Public Health Institutes (NNPHI) | 3 |
| University of Kentucky | 4 |
| University of North Carolina School of Public Health | 1 |
| Emory University | 1 |
| Mathematica Policy Research, Inc | 2 |
| Total | 44 |
Step 3: Conduct an Agenda-Setting Meeting With National Partners, Researchers, and the CDC
In June 2003, a 2-day planning session was convened with the objective of arriving at a consensus-based research agenda. The first day of the meeting was open to all interested CDC researchers, as well as the external partners that had been invited. Approximately 100 people attended. The agenda consisted of presentations by selected researchers on current developments and results associated with public health systems research. All participants were asked to write down research ideas that came to mind during the presentations and submit these ideas to the facilitator (H.T.). Approximately 90 ideas were collected and sorted into the 4 research categories that had emerged from the CDC brainstorming session.
On the second day, 15 CDC staffers met with 29 senior representatives from national partner organizations (Table 1 ▶). Two sequential 45-minute breakout sessions, in which a facilitated discussion of one research category helped to define priorities, were held on each of the 4 research categories. Each participant was given an opportunity to take part in 2 breakout sessions.
After discussion and voting, the 90 research ideas had been narrowed down to 40, which were presented to the entire group for discussion. A nominal group process followed in which each participant was asked to vote for 10 research themes that he or she considered a priority. Although specific criteria were not established, participants were instructed to consider each theme’s public health impact, feasibility, and urgency, as well as whether existing research on the theme was lacking. Of 440 possible votes, 360 (82%) went to 14 of the 40 research themes. When these 14 themes were sorted into the 4 broad research categories previously described, the distribution of votes across the 4 categories was fairly uniform. After the votes were tallied, the group held a facilitated discussion and all participants readily agreed to the final list of 14 priority themes (box this page).
Research Priorities for the Public Health Systems Research Agenda.
Determine how public health agency structure affects performance. (40)
Define and quantify dimensions of public health systems, including interorganizational relationships (including the role of the agency within the public health system). (33)
Explore the relationship between performance and health outcomes (and the chain of impacts that leads from improved performance to improved health outcomes). (30)
Define the characteristics of high-performing local, state, and federal public health agencies. (29)
Explore the relationship between social determinants of health and system performance. (28)
Evaluate the costs of achieving and maintaining acceptable/optimal levels of performance. (This activity includes exploring reasonable models to collect agency financial data.) (27)
Explore the relationship between public health infrastructure/performance and the design, implementation, and impact/outcomes of categorical programs (including the use of evidence-based interventions). (27)
Conceptualize a framework for high-performing public health systems that includes key elements. (26)
Identify, develop, and refine measures of health outcomes that are sensitive to public health systems capacity and performance. (26)
Explore models and outcomes of accreditation of public health agencies and/or public health systems as performance improvement methods. (21)
Evaluate how shifting policy and financial priorities affect performance of public health systems. (19)
Explore what factors and processes facilitate community involvement in using the National Public Health Performance Standards Program in system improvement activities (quality improvement). (19)
Evaluate how and to what extent a high-performing public health system is indicative of preparedness. (19)
Explore the effectiveness (within the agency and the system) of local and state governance structures. (16)
Note. Priorities were established by a group of 44 participants from the Centers for Disease Control and Prevention, national organizations representing public health practitioners, and academic and research institutions. Numbers in parentheses are numbers of votes received.
Step 4: Disseminate the Draft Research Agenda to Interested Public Health Partners for Discussion, Input, and Comment
The results of the 2-day session were presented at the AcademyHealth31 annual conference in June 2003 and at a meeting of the Council on Linkages32 during the joint annual conference of the Association of State and Territorial Health Officials and the National Association of County and City Health Officials in September 2003. The prioritized research themes were presented during 3 research-related sessions at the American Public Health Association annual meeting in November 2003. Comments and suggestions were invited at all these venues.
LESSONS LEARNED
After the agenda-setting meeting, an informal debriefing with key participants indicated a high level of satisfaction with both the process and outcome. The comments and suggestions received when the draft research agenda was presented at the AcademyHealth, Council on Linkages, and American Public Health Association meetings validated the results and encouraged the CDC to proceed with publishing the research agenda and working toward building the necessary infrastructure for public health systems research.
The research agenda points to 3 areas that need to be addressed. (1) There is an immediate need to accurately describe the dimensions of public health systems, including their structure, characteristics, costs, and funding mechanisms, as well as the influence of categorical programs and funding on system performance. (2) Additional research is needed to address the relationship between system performance and such core areas as social determinants of health, public policy, preparedness, and governance structures. (3) There is a need to explore the concepts of performance measurement. Ultimately, the body of knowledge derived from this research will challenge public health leaders, policy makers, and researchers to conceptualize a framework for high-performing public health systems and provide evidence of the impact of system performance on health outcomes.
The comments and feedback we received were similar to those received by others who have developed national research agendas.26–29 What differentiated our efforts from others were the various opportunities for input from the general public health community. Although it is universally acknowledged that a national research agenda is necessary to prioritize and strategically approach public health systems research, we cannot overstate the importance of having gained consensus on this research agenda among national partner organizations, researchers, and the CDC.
THE BENEFITS OF A RESEARCH AGENDA
A comprehensive public health systems research agenda will help funding organizations make informed choices between competing research options. In a time of tightening resources, it is imperative that funders know how to use research dollars to maximum advantage and guide researchers into areas of national interest and priority. A focused and strategic approach to researching public health systems will enable funding organizations to avoid duplication by understanding exactly how their particular efforts integrate with other similar research and how these efforts jointly contribute to developing systemwide knowledge and evidence of performance. A research agenda will give policy-makers evidence they can use to make improvements to deteriorating public health systems that will ultimately result in a higher level of performance and, by extension, improved health in our communities.
The national public health systems research agenda is a consensus-based tool for achieving these goals. With the approach of local and state agency accreditation processes there will be increasing demands for public health systems research. What remains to be done is to develop the resources and infrastructure to support this agenda. Policymakers need to recognize the value of the agenda and provide the necessary resources to support it. We in the public health community need to support the agenda by advocating for resources and marketing the need for and importance of public health systems research. The CDC, Council on Linkages Between Academia and Public Health, AcademyHealth, and other partners must embrace this agenda and provide leadership in developing real and meaningful research funding and supporting infrastructure. Without adequate support and resources, the agenda will fail to achieve its purposes, to the detriment of the nation’s public health system.
Peer Reviewed
Contributors D. Lenaway originated and led the project jointly with P. Halverson. D. Lenaway drafted the original article and coordinated other authors’ contributions. P. Halverson contributed to the writing. S. Sotnikov assisted in the design, implementation, and analysis of the project and in the writing and editing of the article. H. Tilson contributed to the design and implementation of the project, acted as the sole facilitator during the project, and contributed to the writing and editing of the article. L. Corso and W. Millington contributed to the design and implementation of the project and provided assistance in the editing of the article.
References
- 1.Institute of Medicine. The Future of the Public’s Health in the 21st Century. Washington, DC: National Academies Press; 2002. Also available at: http://books.nap.edu/catalog/10548.html. Accessed December 27, 2005.
- 2.Healthy People 2010: Understanding and Improving Health. Washington, DC: US Department of Health and Human Services; 2001. Also available at: http://www.healthypeople.gov/document. Accessed December 27, 2005.
- 3.Mays GP, Halverson PK, Scutchfield FD. Behind the curve? What we know and need to learn from public health systems research. J Public Health Manag Pract. 2003;9(3):179–182. [DOI] [PubMed] [Google Scholar]
- 4.Studnicki J, Steverson B, Blais HN, Goley E, Richards TB, Thornton JN. Analyzing organizational practices in local health departments. Public Health Rep. 1994;109(4):485–490. [PMC free article] [PubMed] [Google Scholar]
- 5.Turnock BJ, Handler A, Hall W, Potsic S, Nalluri R, Vaughn EH. Local health department effectiveness in addressing the core functions of public health. Public Health Rep. 1994;109(5): 653–658. [PMC free article] [PubMed] [Google Scholar]
- 6.Miller CA, Moore KS, Richards TB, McKaig C. A screening survey to assess local public health performance. Public Health Rep. 1994;109(5):659–664. [PMC free article] [PubMed] [Google Scholar]
- 7.Handler AS, Turnock BJ. Local health department effectiveness in addressing the core functions of public health: essential ingredients. J Public Health Policy. 1996;17(4):460–483. [PubMed] [Google Scholar]
- 8.Rohrer JE, Loh SC. Improvements in the performance of local public health agencies. J Public Health Manag Pract. 1999;5(6):55–61. [DOI] [PubMed] [Google Scholar]
- 9.Scutchfield FD, Knight EA, Kelly AV, Bhandari MW, Vasilescu IP. Local public health agency capacity and its relationship to public health system performance. J Public Health Manag Pract. 2004;10(3):204–215. [DOI] [PubMed] [Google Scholar]
- 10.Ford EW, Duncan WJ, Ginter PM. The structure of state health agencies: a strategic analysis. Med Care Res Rev. 2003;60(1):31–57. [DOI] [PubMed] [Google Scholar]
- 11.Turning Point. Survey on Performance Management Practices in States. Available from: Public Health Foundation Training Resource Center, Washington, DC (http://bookstore.phf.org/prod317.htm).
- 12.Mays GP, Miller CA, and Halverson PK. Local Public Health Practice: Trends and Models. Washington, DC: American Public Health Association; 2000.
- 13.Halverson PK. Embracing the strength of the public health system: why strong government public health agencies are vitally necessary but insufficient. J Public Health Manag Pract. 2002;8(1):98–100. [DOI] [PubMed] [Google Scholar]
- 14.Lasker RD and the Committee on Medicine and Public Health. Medicine and Public Health: The Power of Collaboration. New York, NY: New York Academy of Medicine; 1997.
- 15.Halverson PK, Mays GP, Kaluzny AD, Richards TB. Not-so-strange bedfellows: models of interaction between managed care plans and public health agencies. Milbank Q. 1997;75(1): 113–138. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Institute of Medicine. The Future of Public Health. Washington, DC: National Academy Press; 1988. Also available at: http://books.nap.edu/catalog/1091.html. Accessed December 27, 2005.
- 17.Institute of Medicine. Improving Health in the Community: A Role for Performance Monitoring. Washington, DC: National Academy Press; 1997. Also available at: http://books.nap.edu/catalog/5298.html. Accessed December 27, 2005. [PubMed]
- 18.Baker EL Jr, Koplan JP. Strengthening the nation’s public health infrastructure: historic challenge, unprecedented opportunity. Health Aff (Millwood). 2002;21(6):15–27. [DOI] [PubMed] [Google Scholar]
- 19.Halverson PK, Miller CA, Kaluzny AD, Fried BJ, Richards TB, Schenck SE. Performing public health functions: the perceived contribution of public health and other community agencies. J Health Hum Serv Adm. 1996; 4:288–303. [PubMed] [Google Scholar]
- 20.Mays GP, Halverson PK, Kaluzny AD. Collaboration to improve community health: trends and alternative models. Jt Comm J Qual Improv. 1998; 24(10):518–540. [DOI] [PubMed] [Google Scholar]
- 21.Lovelace K. External collaboration and performance: North Carolina public health departments. Public Health Rep. 1996;115(4):350–357. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Halverson PK, Mays GP, Kaluzny AD. Working together? Organizational and market determinants of collaboration between public health and medical care providers. Am J Public Health. 2000;90(12):1913–1916. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Weiss ES, Anderson RM, Lasker RD. Making the most of collaboration: exploring the relationship between partnership synergy and partnership functioning. Health Educ Behav. 2002;29(6): 683–698. [DOI] [PubMed] [Google Scholar]
- 24.National Public Health Performance Standards Program. Available at: http://www.cdc.gov/od/ocphp/nphpsp. Accessed December 27, 2005.
- 25.Turning Point. Offering resources and partnerships to improve public health. Available at: http:/www.turning-pointprogram.org. Accessed December 27, 2005.
- 26.Hartley D, Britain C, Sulzbacher S. Behavioral health: setting the rural health research agenda. J Rural Health. 2002;18(suppl):242–255. [DOI] [PubMed] [Google Scholar]
- 27.Cooper JK, Clancy CM. Health services research agenda for clinical preventive services. Am J Prev Med. 1998; 14(4):331–334. [DOI] [PubMed] [Google Scholar]
- 28.Dannenberg AL, Jackson RJ, Frumkin H, et al. The impact of community design and land-use choices on public health: a scientific research agenda. Am J Public Health. 2003; 93(9):1500–1508. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Cioffi JP, Lichtveld MY, Tilson H. Research agenda for public health workforce development. Available at: http://www.phf.org/Link/Workforce_Agenda_Poster.pdf (PDF file). Accessed December 27, 2005. [DOI] [PubMed]
- 30.Public Health in America. Washington, DC: Public Health Functions Steering Committee, US Public Health Service; 1994.
- 31.AcademyHealth: advancing research, policy, and practice. Available at: http://www.academyhealth.org. Accessed December 27, 2005.
- 32.Council on Linkages Between Academia and Public Health Practice. Available at: http://www.phf.org/Link.htm. Accessed December 27, 2005.