Baseline Assessment of Public Health Informatics Competencies in Two Hudson Valley Health Departments

SYNOPSIS

Information technology has the capability to improve the way public health is practiced. Realization of this potential is possible only with a workforce ready to utilize these technologies. This project team assessed informatics competencies of employees in two county departments of health. The goal was to determine the status quo in terms of informatics competencies by surveying current levels of proficiency and relevance, and identify areas of needed training. A survey was adapted from the recommendations of a Working Group document by the Centers for Disease Control and Prevention and administered to all employees in the two health departments. Respondents evaluated proficiency and relevance for each of 26 recommended competencies. A gap score was generated between these two measures; results were compared to the recommendations of the Working Group.

The following data for each job level are presented: mean gap scores by competency class; the percentage of respondents demonstrating a gap in the competencies reported to be most relevant; and the percentage of respondents meeting the target recommendations of the Working Group. The percentage of respondents who reached the targets was low in higher-level staff. And overall, employees reported low levels of relevance for most of the competencies. The average public health employee does not feel that prescribed informatics competencies are relevant to their work. Before the public health system can take advantage of information technology, relevant employee skills should be identified or developed. There needs to be a shift in thinking that will recognize the promise of information technology in everyday work.

Much has been written about the potential of information technology to vastly improve the capabilities of the public health workforce. Individual systems are being used in local public health settings for surveillance, vital statistics, immunization registries, and emergency preparedness functions, but there is little quantitative data regarding the extent of informatics proficiency in the public health workforce. Public health departments need staff ready and proficient to perform in an information society—able to effectively use information and information technology, and to manage information technology projects. With scarce resources, empowering existing staff with the skills necessary to perform in the constantly changing environment is critical. This study sought to determine the status quo in terms of informatics competencies by measuring current levels of proficiency and to identify areas of needed training.

In 2000, Yasnoff defined public health informatics as “the systematic application of information and computer science and technology to public health practice, research, and learning.”¹ In 2002, a Working Group of 45 experts led by Patrick O'Carroll and supported by the Centers for Disease Control and Prevention (CDC) brought national attention to the subject. This Working Group produced a consensus document, “Informatics Competencies for Public Health Professionals.”² The identified competencies, or “observable measures of skills,” were designed to complement the core competencies defined by the Council on Linkages Between Academia and Public Health Practice.

For each of the 45 informatics competencies, a proficiency target was recommended by job level. Since publication, there has been no identified attempt to apply these recommended public health informatics competencies as a measure of skills in the public health workforce. Furthermore, there has been no evaluation of educational interventions designed to improve these competency levels. This article reports the results of the application of an instrument designed to assess current levels of informatics competency in the public health workforce. This study represents the first formal collaboration between New York Medical College and Hudson Valley county health departments—the Public Health Information Partners (PHIP).

BACKGROUND

Healthy People 2010 affirms the need for a public health workforce that is able to use information technology effectively for networking, communication, and access to information.³ Reports from the Institute of Medicine (IOM) highlight the continued “disarray” of the nation's public health system as well as the need for a well-trained public health workforce.^4,5 The 1997 report, “The Public Health Workforce: An Agenda for the 21st Century,” confirms the need for a stronger public health workforce along with a data system to track the extent to which the workforce has the knowledge, skills, and abilities to carry out their functions.⁶ Yet by 2003, it was reported that public health continues to be “plagued by an underuse of information and technology, and the literature regarding public health information needs and information-seeking behavior is still in its infancy.”⁷

In 1998, The New York Academy of Medicine sponsored a national forum that highlighted the information needs of the public health workforce.⁸ The forum was the first major event to focus on the diverse information needs of the public health workforce, but did not address the competency level of the workforce in the area of information technology. Following this forum, the Academy and the National Library of Medicine held a session that resulted in a strategic plan; one of the goals is “to train the public health and health policy workforce to use information resources effectively and provide them with the support of skilled librarians ….”⁹

In 2001, the National Association of City and County Health Officials (NACCHO) produced its “Chartbook,” which is one of the few national sources of data on local public health agencies (LPHAs) in the United States.¹⁰ Respondents to the NACCHO survey named their top two training needs: job-specific and information technology. By June 2005, NACCHO distributed its fourth national survey to every local public health agency in the United States, and for the first time included a few questions related to informatics.¹¹

In 2003, Lee, Giuse, and Sathe benchmarked the information needs of public health personnel from county health departments in Tennessee.⁷ Their findings confirmed the “overarching theme of the information crisis in public health with significant diversity in education, job functions, computer access, and technology skills.” This study was the first step in developing a training partnership with public health workers throughout Tennessee. This model matches most closely the broad mission and objectives of our Hudson Valley project.

In 2004, the Public Health Informatics Institute and NACCHO studied the need for public health informatics training in terms of “putting training on track.”¹² Respondents in this survey reported the gap between the need and demand for informatics training, as well as a lack of resources to meet perceived informatics training needs. They concluded that “informatics competencies have not been adequately assessed among local public health agency personnel, have not been given the priority they needs, and training is often unavailable in a format that meets the agency's needs, time frames, and cost constraints.”¹²

METHODOLOGY

This study focused on employees of the Orange and Putnam County departments of health, the second-largest and second-smallest counties in terms of population in the Hudson Valley Region of New York State (341,376 and 95,745, respectively [2000]).¹³ All 235 employees of both counties (161 in Orange County and 74 in Putnam County) were invited to participate in the study.

The PHIP survey instrument was adapted in part from the aforementioned Working Group document, “Public Health Informatics Competencies.”² The report grouped 45 public health informatics competencies into three broad classes: (1) the use of information per se for public health practice, (2) the use of information technology to increase individual effectiveness, and (3) the management of information technology projects to improve the effectiveness of the public health enterprise.³ For each competency, five increasing expertise levels were recommended ranging from “aware” to “proficient” for each of the job levels defined by the Council on Linkages: frontline, senior (senior-level technical staff), supervisory (supervisory and management staff), and, for some competencies, clerical.² According to the Council, frontline staff carries out the daily functions of the health department; senior-level staff functions with specialized skills and/or technical knowledge in such areas as epidemiology and program planning; and supervisory staff functions at a high level of knowledge and skills, often with staff reporting to them. The latter are generally called upon to speak at public functions, manage sizable projects, and make recommendations on policy issues. Finally, clerical staff provides basic staff support to the other levels.

Twelve competencies from the original 45 in the O'Carroll consensus document were selected as questions for the survey, based upon whether or not training was within the scope of PHIP project staff expertise and, therefore, financially feasible given the project budget. In addition, 14 recommended “learning objectives” from the document were used as questions. For each question, participants were asked to report their level of proficiency on a five-point scale and the relevance to their work on a three-point scale. For a complete listing of the competencies used, refer to Table 3.

Table 3.

Staff public health informatics competencies mapped by job levels to target levels^a

^aTargets are the recommended levels (numerically coded) posited by the O'Carroll document (0 = not aware; 1 = aware; 2 = aware to knowledgeable; 3 = knowledgeable; 4 = knowledgeable to proficient; 5 = proficient).

The data collection instrument included three components: job-level designation, public health competency survey, and an information needs assessment. The instrument was tested with a cohort of graduate students in the School of Public Health. Based on this validation, the instrument required 11 to 20 minutes to complete. The instrument was administered on-site to local health department employees from November 16, 2004, through January 20, 2005, and distributed immediately following general orientation sessions at the county offices. Anonymity was assured through a unique coding system and incentives were provided to encourage completion of the entire instrument. Data analysis was conducted using Stata statistical analysis software.¹⁴

The focus of the study was to determine proficiency levels on a series of recommended competencies. In addition, the survey questioned the relevance of each competency to respondents' job responsibilities. A “gap score” was generated by subtracting the proficiency score from the relevance score. Logically, if a competency is considered relevant to one's work, a matching proficiency level should be expected. Gaps between proficiency and relevance indicated a deficit and were to be considered opportunities for training. A gap score equal to zero suggests a match between relevance and proficiency. The five-point proficiency scores were rescaled to enable this gap score calculation.

Preliminary results were presented and discussed with Orange and Putnam County officials in March 2005, with follow-up in June 2005.

RESULTS

A total of 235 surveys were distributed. Of the 227 completed surveys, 211 surveys could be coded by job-level designation for later analysis. Respondents indicated their job levels based upon the definitions from the Working Group document, including: 86 frontline staff (40.8%), 26 senior staff (12.3%), 47 supervisory staff (22.2%), and 52 clerical staff (24.6%). The job-level distribution of respondents by county is indicated in Table 1. The distributions by job level for the two counties are proportionately similar.

Table 1.

Respondents by county and job level

NOTE: A total of 235 surveys were distributed (161 in Orange County, 74 in Putnam County), and 227 were completed (153 in Orange County, 74 in Putnam County). The difference is due to missing job-level data.

The Figure depicts mean gap scores by job level for each competency class: Class 1 = use of information; Class 2 = use of information technology; and Class 3 5 management of information technology projects. The supervisory-level data is consistent with higher expectations and demonstrates adequate mean proficiencies relative to relevance means in Class 1 competencies, and to a certain extent in Class 3, but demonstrates a clear deficiency in Class 2. The senior staff level, as might be expected, demonstrates adequate proficiency in Classes 1 and 2 relative to reported relevance, but indicates needed training in Class 3. Finally, frontline staff demonstrates basic proficiencies in Class 1, but appears to need training in Classes 2 and 3. In aggregate, clerical staff reports levels of knowledge and proficiency above their relevance scores, indicating that no additional training is needed to carry out job functions. In addition, the relevance scores for the clerical staff were low.

Figure.

Orange and Putnam County mean gap scores by class

Previous documents report competencies by broad class, such as the O'Carroll document and recent NACCHO study noted previously.^2,12 However, analysis by broad class alone does not provide enough specific information to identify potential deficits that would lead to needed educational interventions. For this reason, data at the individual competency level, as opposed to aggregated job-level data, is the focus of this study.

Table 2 presents in rank order those competencies that received at least a mean relevance score of 1.8 (1=somewhat relevant; 2=relevant) for each job level. For each of these competencies identified as most relevant, the percentage of respondents with a gap score of ≥1 (and number of employees this percentage represents) is presented. Clerical staff is not represented on this table as none of the competencies had a mean relevance of at least 1.8.

Table 2.

Competencies reported to be most relevant with associated percent and number of respondents indicating a gap ≥1

^aRelevance scoring: 0 = not relevant, 1= somewhat relevant, 2 = relevant, 3= highly relevant

The complete list of surveyed competencies is included as Table 3. The target levels are the recommended proficiency levels from the Working Group document, numerically coded to match the values used in the survey. Table 3 shows the number and percentage of respondents who either met or exceeded the targets. Because most of the competencies for the clerical level were not designated, the data are not included in Table 3.

DISCUSSION

The results provide valuable information regarding the readiness of public health workers to use information and information technology, and to manage information technology projects. In short, not only do the results show a gap between the recommended and reported levels, they reveal that the public health workers in these two counties on average don't recognize most of these competencies as relevant to their day-to-day functions in the health department. However, as will be reported elsewhere, respondents were extremely interested in training on specific software programs and topics.

On average, most of the Class 2 and 3 questions were not viewed as relevant by any job level. This made the identification of areas for needed training complex. For example, the question regarding the ability to “design an agency/organization website” (#14) had a large number of respondents who had a sizeable gap score, and hence was originally thought to be a potential area for training. However, when coupled with mean relevance, it disappeared as a priority. In other words, a few people thought website design might be a usable skill for them in their work at the department, and knew nothing about it, but on average, most employees found it to be completely irrelevant. Furthermore, conversations with key staff at the health departments revealed that most website design is carried on outside the department.

While the majority of the frontline staff met or exceeded the targets in Class 1, but less so in Classes 2 and 3, the senior and supervisory staff are expected to be “proficient” in all three Classes, but fell short. In addition, measures of relevance in most areas appear to indicate an overestimation on the part of the consensus document recommendations. However, it must be concluded that the recommendations are forward-thinking in their conception, and that status quo perceptions are not yet synchronous with the vision of the experts on the CDC Working Group.

There were some methodological challenges and limitations to this study. First, refining the survey instrument was an iterative process that required rewording and reformatting to reduce the visual complexity and length. The wording of some of the original competencies was academic and judged by key county staff as too vague for public health workers to understand. As one commissioner noted, “‘nascent’ and ‘essential underpinnings’ are not words that my staff would use.” Second, as with all surveys of this nature, the PHIP data collection instrument is subject to self-reporting biases. Third, it would be optimal to include all of the original 45 competencies in the survey. And lastly, the need to rescale the proficiency scale to enable comparison with the relevance score was an unfortunate design inconvenience.

The overall strength of this pilot survey is that it was a grassroots study completed in the field with an overall 97% response rate. The commitment of the local health department commissioners lent credibility to the importance of the study that embraced all staff at all levels. Feedback from the local health department leadership of both counties was overwhelmingly positive. One commissioner reported that, “Staff enjoyed the process, especially in the middle of the winter, and by involving all staff it was positive reinforcement of the fact that everyone was part of the public health team and plays a valuable role.” The opportunities to work with an academic institution offering assistance in areas that were believed to be weak was “an opportunity not to miss.”

With educational interventions identified in the next phase, the initial concerns that the study was merely an academic exercise were addressed. Again, a commissioner noted, “This project will open a whole new world for a lot of staff who do not understand how much information is truly available to them.” From the New York Medical College School of Public Health (SPH) and Health Sciences Library (HSL) viewpoints, this collaboration melds the education and information support with on-site training. The SPH supports local health professionals with a commitment to provide lifelong learning opportunities for all public health professionals. The HSL supports the needs of the faculty and the students, who will soon be alumni and health professionals in the field. The survey outcomes will help assure that the SPH curriculum meets the needs of the public health workforce.

CONCLUSION AND NEXT STEPS

The purpose of this study was to determine the status quo, to measure levels of informatics awareness, and to identify areas for needed informatics training. The results provide baseline data, which will be used to assess outcomes. It is not expected at this point that outcomes will be substantial for all competencies, but by targeting select groups and customizing educational interventions, there may be select impacts in line with health department goals. One measure of success thus far is that one additional Hudson Valley county has agreed to participate in the PHIP project as part of a subcontract to this National Network of Libraries of Medicine project.

Currently, training is underway, designed based on the results presented here, and through conversations with county health commissioners and key staff. In general, training is currently focusing on three areas: finding and presenting information and data; using Microsoft® Office software for information communication and presentation; and obtaining grants—finding information and making it happen.

The results of the PHIP project point to substantial discrepancies between the recommended informatics proficiency levels and what working public health professionals view as relevant to their specific job functions. The average employee in local public health departments does not feel that prescribed informatics areas are relevant to daily work. Before the public health system will be able to take full advantage of information technology, relevant employee skills should be identified or developed. In addition, there needs to be a shift in thinking at all levels of public health organizations, which will recognize the promise of information technology in the everyday work of all employees. Only then will employees recognize the relevance of information, information systems, and information technology projects to their public health function.