Development and Implementation of a Public Health Workforce Training Needs Assessment Survey in North Carolina

SYNOPSIS

Assessing the training needs of local public health workers is an important step toward providing appropriate training programs in emergency preparedness and core public health competencies. The North Carolina Public Health Workforce Training Needs Assessment survey was implemented through the collaboration of several organizations, including the North Carolina Center for Public Health Preparedness at the North Carolina Institute for Public Health, the outreach and service unit of the University of North Carolina School of Public Health, the Office of Public Health Preparedness and Response in the North Carolina Division of Public Health Epidemiology Section, and local health departments across the state.

A 2003 report from the Institute of Medicine (IOM), Who Will Keep the Public Healthy? Educating Public Health Professionals in the 21st Century, reinforced the call to train the public health workforce—specifically those who work in public health departments or agencies—in core public health skills.¹ Over the last decade, several committees, agencies, and reports have identified public health worker training as a critical component of a well functioning public health system.^1–5

It has been well established that most public health agency employees need training in core public health competency areas.^1–3 In fact, many public health agency workers enter the public health workforce with training in only their specific technical area (e.g., health education, environmental health, or nursing).² While technical expertise is critical, basic awareness and understanding of core public health competencies is also imperative for a public health agency to function well.

Through a national process that included input from public health practitioners, the Council on Linkages Between Academia and Public Health Practice developed a list of core public health competencies, which are organized within the 10 essential public health services (Figure 1). The Council recommends that all public health workers have at least a basic awareness of these competencies, and depending on the competency area, that workers in specific jobs should attain defined skill levels and/or proficiencies. Public health trainers, federal agencies, and academics have utilized these competencies to plan and implement professional development activities in public health.⁶

Figure 1. The 10 essential public health services.

SOURCE: Public Health Functions Steering Committee. Essential public health services. Adopted 1994. Also available from: URL: http://web.health.gov/phfunctions/public.htm

Because there is little consistency in public health worker job definitions among state and local governments across the country, it has been—and remains—difficult to establish the size and composition of the public health workforce.³ For example, North Carolina has a state public health agency and 85 independent local health departments. Among these entities, there are more than 150 different job titles or classifications for the public health workforce. The Bureau of Labor Statistics has worked to establish standard occupational classifications for public health.⁷ Standard occupational classifications should reduce inconsistency in job titles and improve the understanding of the workforce composition and the ability to plan workforce training initiatives.

IMPORTANCE OF ASSESSING THE PUBLIC HEALTH WORKFORCE

The IOM report highlighted the need to assess the public health workforce, emphasizing that the training, education, and competency of public health workers are critical components of public health infrastructure:

The issue of workforce training and competency is central to the success of any public health system. Governmental public health agencies have a responsibility to identify the public health workforce needs within their jurisdictions and to implement policies and programs to fill those needs. In addition, an assessment of current competency levels and needs is essential to develop and deliver the appropriate competency-based training, as well as to evaluate the impact of that training in practice settings.⁴

Given the large number of competencies (60–68 total), assessing the workforce is an efficient approach to prioritizing training activities. Recently, public health researchers have explored two methods of surveying the public health workforce to identify competency training needs. The first surveys all public health workers in all competencies. This method follows the rationale that all workers should have at least a basic awareness of all core competencies. The second method identifies competencies that are important to specific job classifications, asks the worker to self-identify a job classification, and then surveys the worker in the competencies important to that classification. This method reduces the time burden on the worker to complete the assessment, yet also presupposes that the worker will not need awareness of or training in competencies not included in a specific job classification.

Both methods survey each worker within an agency rather than surveying only a subgroup or sample of workers. This results in improved understanding of the composition of the workforce (e.g., worker demographics, job classification, prior training) and provides individualized training profiles. To achieve professional development recommendations, each public health worker should have access to an individually tailored training plan.^1,4

NORTH CAROLINA PUBLIC HEALTH WORKFORCE ASSESSMENT

The North Carolina Institute for Public Health, the outreach and service unit of the University of North Carolina at Chapel Hill School of Public Health (UNC SPH), has a long history of providing continuing education to public health workers in North Carolina. The school's Office of Continuing Education conducts extensive training for environmental health specialists, nurses, and social workers through state-of-practice committees for each of these professions. The state-of-practice committees are comprised of practitioners, academics, and leaders who identify training needs that best meet the certification requirements of these professions. The Office of Continuing Education has for many years provided training on topics of interest to North Carolina public health leaders and other workers. Nonetheless, prior to the establishment of the North Carolina Center for Public Health Preparedness (NCCPHP) in 2000, there had been no organized effort to provide professional development plans to all North Carolina public health workers regardless of occupational classification, particularly in the emerging area of public health preparedness.

NCCPHP is part of a nationwide network of Centers for Public Health Preparedness funded by the Centers for Disease Control and Prevention (CDC) to improve the capacity of the public health workforce to prepare for and respond to terrorism and other emerging public health threats. The NCCPHP is charged with assessing state and local public health workers and providing training in emergency preparedness and response, bioterrorism, and core public health skills. To achieve these objectives and deliver innovative, effective training opportunities, the NCCPHP develops partnerships with local and state health organizations.

Survey development

The NCCPHP designed the Public Health Workforce Training Needs Assessment survey to collect information for public health agencies and to identify staff training priorities for individual public health workers, each local health department, and the state public health workforce. The survey instrument was designed to allow public health workers to self-assess their level of need for training on all core public health competencies. A similar instrument used by Tulane University served as a reference for the questionnaire design.⁸

The survey contained a list of job activities based on the core public health competencies organized by the 10 essential services. Individuals self-assessed the importance of each activity to his or her job on a scale of 1 (not at all important) to 4 (very important). Then, individuals rated personal need for training to perform this same activity on a scale from 1 (no need for training) to 4 (very high need for training). The survey also contained questions about previous education and training, level of interest in future training and education, and barriers and facilitators to training. It also included several questions about worker access to and use of computers and the Internet to determine the feasibility of online training courses. After developing a draft survey, NCCPHP requested review and feedback from public health workers in a local health department. Based on their responses, items were reworded for clarity and the final survey was prepared for pilot testing.

NCCPHP conducted a pilot test of the survey in three local health departments in North Carolina from May through September 2001. Gaining the cooperation and support of local health department managers was an important first step in maximizing employee participation and survey response. NCCPHP staff first met with state and local health department management, then NCCPHP staff and health department management briefed health department employees on the survey, its purpose, and employee rights in research involving human subjects. All employees of the three health departments in the pilot study phase, including administrative support staff, were asked to complete a paper copy of the training needs assessment. NCCPHP assigned unique identifiers to each survey and health department staff members were provided with addressed envelopes to return surveys confidentially. Survey data were reported only in aggregate by organization or occupational classification and no individual data were released. The UNC Public Health Institutional Review Board approved the protocol and instrument for the initial pilot questionnaire and subsequent versions of the survey.

The combined response rate from three participating counties in the pilot study phase through September 2001 was 67% (range: 57% to 78%) of more than 800 public health workers. To achieve this response rate, NCCPHP staff sent multiple reminders via e-mail, mail, and telephone to health department staff members to encourage survey completion.

The accuracy of pilot study findings was determined by examining survey results for workers in specific job classification categories established by the Public Health Standard Occupational Classification System from the Bureau of Labor Statistics.⁷ Representatives from the North Carolina state-of-practice committees for social work, nursing, and environmental health examined survey results for their respective professions and indicated that the priority training needs identified by the survey matched closely with the needs identified through the state-of-practice committees.

Although the pilot survey achieved a moderate response rate, implementation methods were costly in terms of personnel time, including creating and disseminating follow-up reminders, creating a data entry database, and hand-entering data. Administering the survey to the entire local public health community in North Carolina was necessary to meet requirements of the state bioterrorism supplemental grant.⁹ To facilitate data collection for the thousands of North Carolina public health workers who would be surveyed, NCCPHP worked with the UNC SPH Instructional and Informational Systems unit to create an Internet-based version.

Questions on emergency preparedness and response were included in the survey (Figure 2) to address the need to improve public health capacity in bioterrorism and other public health emergencies. These questions were based on the emergency preparedness competencies published by the Center for Health Policy at the Columbia University School of Nursing.¹⁰

Figure 2. Emergency preparedness competencies.

SOURCE: Adapted from “Core public health worker competencies for emergency preparedness and response,” Columbia University School of Nursing, Center for Health Policy.

The emergency preparedness and response competencies were included in the survey in a format similar to that of the core public health competencies, except that individuals were asked to rate their confidence to carry out the activity on a scale from 1 (not at all confident) to 4 (very confident) rather than rating the importance of each activity to his or her job. This question was changed for the emergency preparedness section because all public health employees must be prepared to respond appropriately in an emergency, regardless of their normal job duties. Individuals were also asked to rate personal need for training to perform the activity on a scale from 1 (no need for training) to 4 (very high need for training), as in other sections of the survey.

NCCPHP also shortened the survey based on responses from pilot test participants, reducing the number of questions regarding employee access to computers and the Internet and deleting sections on training barriers, as little response variation was found in these items.

Survey implementation

Approach.

In 2002, as part of a national response to terrorism activity, the Office of Public Health Preparedness and Response in the North Carolina Division of Public Health collaborated with NCCPHP to implement the training needs assessment survey with local public health workers throughout the state. This collaboration was essential to the implementation process and to achieving acceptable response rates. Local health department staff members are frequently asked to complete surveys, so it was important to recognize the limited amount of time they have available to participate. Additionally, public health department staff members have many and varied responsibilities, so it was helpful to offer a valuable incentive for participation, such as personalized training plans for individuals and organizations.

Methods.

NCCPHP implemented the statewide online survey in four tiers. This staged approach was designed to provide a systematic way to implement the survey to every public health worker in the state without exceeding NCCPHP's capacity to provide information and technical support. The Table presents a summary of the implementation process.

Table. North Carolina public health workforce survey implementation process.

NCCPHP first provided the online survey to 38 members of seven new Public Health Regional Surveillance Teams (PHRSTs) from August through October 2002. From October through December 2002, NCCPHP surveyed 752 members of local health department Epidemiology Teams identified by local health directors. Survey implementation began with PHRSTs and Epidemiology Teams to provide each health department with a local point of contact familiar with the computer system and able to assist others logging into the system for the first time.

The public health workforce in 23 of 85 local health departments was surveyed from November 2002 through May 2003. After completion of Tier III data collection, NCCPHP summarized survey results to establish preliminary training needs and provided a report of those needs to state and local officials.

In June 2003, the survey was incorporated into an integrated online learning management system called the Public Health Workforce Development System (PHWDS), which connects individuals' assessed needs to competency-matched trainings. Tier IV implementation was conducted using the PHWDS, and data collection continued through October 2004.

As in the pilot study, successful implementation of the survey required partnership with local health department leaders. Survey implementation began through discussions with health department directors, supervisors, managers, bioterrorism coordinators, administrative assistants, computer personnel, and others in each local health department. NCCPHP also formed partnerships with PHRST members throughout the state who are familiar with key contacts in each health department. PHRST contacts were a valuable asset to both the initial contact to local public health workers and the follow-up process.

Given the survey's multiple phases of implementation, an effective marketing plan was necessary to disseminate information about the survey and encourage response. The marketing plan included presentations, demonstrations, memos, letters, telephone calls, e-mails, posters, and fliers. To encourage response during the first three tiers of the survey, NCCPHP created informational and reminder letters and memos on letterhead from the Office of Public Health Preparedness and Response. Reminder letters were mailed to management and key contacts in each local health department approximately every other month. Initial response rates during Tiers II and III were quite low, and multiple reminders to non-responders were needed. For these tiers, NCCPHP staff conducted all follow-up to non-responders, including up to three personal phone calls to each potential participant over a six-month period. At that time, the Office of Public Health Preparedness and Response encouraged but did not require local health department participation. Local health department employees were contacted over time using a variety of formats to effectively communicate which agencies were initiating the survey, why data collection was important, how to complete the assessment, how data would be used, and the importance of participation by all employees in local health departments.

Beginning in fall 2003, the Office of Public Health Preparedness and Response initiated a requirement that 75% of employees in each local health department complete the online survey by September 30, 2004, in order for local health departments to receive state bioterrorism funds. This additional incentive resulted in a significant increase in survey completion across the state between January and September 2004.

Response rates

PHRST members, Epidemiology Team members, and participants in 23 local health departments (see Tier I–Tier III in the Table) participated in the online survey between October 2002 and May 2003. Response rates were 95% for PHRST members (36 of 38), 67% for Epidemiology Team members (507 of 752), and 42% for the cohort of local health department employees (650 of 1,560). Survey data collection in all local health departments in North Carolina (Tier IV) continued until October 1, 2004. In total, 78% (6,482 of 8,306) of North Carolina public health workers completed the online survey. Data from the online survey identified priority training needs for each group in emergency preparedness and response competencies and core public health competencies.

DISCUSSION

The 78% response rate to date in Tier IV of data collection (June 2003–October 2004) was the result of extensive work by four full-time-equivalent NCCPHP staff, in addition to the Office of Public Health Preparedness and Response requirement that 75% of workers in each local health department complete the survey in order to receive bioterrorism grant funding. NCCPHP staff members devoted considerable amounts of time to facilitating survey implementation and addressing barriers to survey response.

Considering the limited time and budgets available in local health departments, it is imperative to offer efficient and easily accessible assessment options. An initial barrier to the survey implementation was the time needed to complete the survey (30–40 minutes). To address this barrier, the online survey was designed so individuals could complete separate sections over time instead of all at once. For instance, an individual can complete one or two short sections online, save his or her answers, and return as many times as needed to complete the survey. In addition, the survey can be accessed online from any location at any time of the day or night.

Another important consideration was that health department employees are frequently asked to complete surveys by local, state, and national entities, and must be presented with convincing reasons to spend time and energy away from their other duties. In many cases, when an academic institution is involved with a survey process, employees may immediately perceive that they are being asked to help with an outside agency's research and that there is little benefit to the individual or their workplace. In this case, health department workers expressed confusion during Tier III regarding survey sponsorship. This confusion was understandable due to the number of surveys health department staff are asked to participate in by multiple entities over time and also because this survey was sponsored by both the state and a familiar academic unit.

To address these issues, NCCPHP developed a comprehensive marketing plan designed to communicate that the survey was initiated by the state Office of Public Health Preparedness and Response, that NCCPHP was a partner in the effort to fulfill state and national workforce requirements, and that survey data would be used to create training plans for individuals, occupational groups, and agencies. Fliers, posters, letters, e-mails, and presentations to local health department staff members and administration emphasized the immediate and long-term benefits to individuals and agencies if they participated. Communications during Tier IV focused on the survey's ability to offer direct feedback to the participant once the survey was submitted online, which is an innovative feature. Beginning with Tier IV, the survey is now part of a system that goes beyond merely assessing the workforce's knowledge gaps in emergency preparedness and core public health competencies; it immediately creates an individualized training plan by prioritizing the competencies based on an individual's answers to the survey, then links individuals to competency-matched trainings.

Since January 2004, marketing efforts also included clear communication about the 75% response rate requirement for local health departments to receive state bioterrorism funds. However, communicating these messages while concurrently providing instructions on how to log in and begin the online survey proved to be a great deal of information to relay at one time. These messages (e.g., how and where to log in and get started, why this effort is important, why the survey is tied to funding from the state, who is sponsoring the survey) had to be prioritized and delivered separately over time.

A significant barrier to implementation of the online survey was limited computer access and computer literacy. Despite increased computer access across North Carolina over the past decade and recent installation of T-1 lines (high-speed Internet access) in all 85 local health departments, computer access limitations still exist. Some health departments have one computer to serve an entire department or group of employees, so only shared workstations are available in many cases. Also, the digital literacy of public health workers varies considerably. Many local health departments have computerized systems that workers must use regularly, but others do not use computer technology to conduct daily business or clinical services. For workers who are located in clinics or conduct home visits, regular computer use is not imperative; they may not have access to a computer at work or at home.

To assist workers with entry-level computer literacy, NCCPHP provided on-site assistance to health departments through presentations and demonstrations of the online survey. In more than 50 local health departments, NCCPHP staff members demonstrated the survey at a staff meeting or forum chosen by the health department management, and remained after the demonstration to provide individual assistance with the log-in process. NCCPHP supplied a laptop computer for these assistance sessions if necessary. Of all the outreach and communication activities implemented, the in-person demonstrations were most effective in improving the response rate. Also helpful were clear, concise, printed instructions for people with varying levels of computer skills and creative one-page printed handouts that provided an overview of the survey, its functions, and its usefulness to individuals and organizations. To further address computer literacy and access issues, paper copies of the survey were always available upon request and responses were input by NCCPHP staff members.

Most health directors and supervisors are eager for their staff members to receive training needs assessment, training plans, and competency-matched trainings. However, another barrier to survey implementation was apprehension by some health directors and supervisors that employees who receive additional training and become more skilled in their jobs may seek employment elsewhere. This is primarily a concern for small health departments that may lose employees to larger health agencies that can offer higher pay and more opportunities for advancement. For this survey, however, most barriers to implementation appeared to be outweighed by the Office of Public Health Preparedness and Response requirement that 75% of employees in all local health departments must complete the survey to receive state bioterrorism funds.

Valued longtime partnerships, consistent communication, and strong support among NCCPHP, the North Carolina Division of Public Health, the University of North Carolina at Chapel Hill School of Public Health, the North Carolina Institute for Public Health, and local public health departments across North Carolina has been and continues to be a significant reason for the success of this assessment effort.

FUTURE DIRECTIONS

Through 2004, the North Carolina Public Health Workforce Development Assessment survey primarily focused on baseline data collection for public health workers. With accreditation of public health agencies becoming a statewide and national consideration, there is increasing interest among health departments to pursue individual and organizational training plans. NCCPHP and the Instructional and Information Systems unit in the UNC School of Public Health established the online Public Health Workforce Development System to directly link individual survey results with applicable trainings. This learning management system, which contains the Public Health Workforce Training Needs Assessment survey instrument, became available to all public health workers in North Carolina in June 2003 with Tier IV survey implementation.

NCCPHP has used preliminary findings from the assessment to guide the development of online training modules and training programs. For example, numerous epidemiology and biological agent training modules have been created for the NCCPHP website. Final survey findings will inform the development, improvement, and maintenance of training programs, and will provide data to create customized training plans for local health departments.

NCCPHP will continue to monitor the training needs and track the progress of public health workers in North Carolina. These efforts will help ensure the development and dissemination of effective training options and will work toward improving the awareness, knowledge, and skill levels of public health workers in emergency preparedness and response and core public health competencies.

In the opening pages of the IOM report, The Future of the Public's Health in the 21st Century,⁴ a quote attributed to Goethe reads: “Knowing is not enough; we must apply. Willing is not enough; we must do.” This quote is a valuable insight into the IOM's recommendations. Assessment of knowledge will continue to be important as we move ahead in public health in the 21st century, and the NCCPHP continues to inform public health workers that an integrated online workforce development system is available for a multitude of uses. NCCPHP looks forward to ongoing participation, improving online public health trainings, offering customized training programs for individuals, and improving knowledge-based planning to address top priorities for organizations, geographic regions, and occupational classifications of the public health workforce in North Carolina.