Development and Scoring of a Survey on Public Health Accreditation Capacity

Abstract

Objectives:

Public health accreditation is a 7-step process that starts with a period of preapplication during which a health department assesses its readiness for accreditation. However, no tools with established reliability and validity that quantitatively measure a local health department’s (LHD’s) capacity for accreditation are available to complete this initial step. We developed and validated a survey to measure accreditation capacity for LHDs.

Methods:

From January through April 2015, we administered a cross-sectional electronic survey instrument with 15 questions that tapped into domains of capacity for public health accreditation. We analyzed and grouped responses by using a confirmatory maximum likelihood factor analysis with oblique rotations. We assessed reliability by using Cronbach α, and we assessed validity by comparing responses with previously established instruments. We administered the survey to 174 LHD directors in Colorado, Kansas, and Nebraska, 153 (88%) of whom responded.

Results:

The factor analysis produced a 3-factor model of accreditation capacity, suggesting that accreditation capacity depends on 3 distinct latent constructs: support for accreditation, preparation, and planning and approach. The model had good scale reliability (average Cronbach α = 0.7) and validity (average factor correlation = 0.43).

Conclusions:

The survey developed and scored in this analysis can be used by LHDs to inform the feasibility of initiating the time-intensive and costly process of accreditation.

For more than a century, efforts have been made to standardize and evaluate the delivery of public health services. In 1914, Public Health Reports described the need for standard practices in public health.¹ However, it was not until 2003 that the Institute of Medicine called for a national discussion on whether a public health accreditation program would be desirable or even feasible.² The result was a 2006 publication that described a public health accreditation model.³ This model evolved into the Public Health Accreditation Board (PHAB; http://www.phaboard.org), the first board responsible for orchestrating a public health accreditation process.

PHAB launched its voluntary public health accreditation program in 2011, making public health accreditation a national program that aligns a health department’s performance to nationally recognized, evidence-based standards.⁴ The accreditation program provided local health departments (LHDs) with a framework to explore opportunities for improvement, improve support for accreditation, and promote connections in their community. The benefits of public health accreditation include improving the quality of public health services, standardizing public health activities, engaging in continuous quality improvement, and helping secure funding.

Although accreditation has many potential benefits, the public health accreditation process can be time-consuming and costly.⁵ A growing body of literature supports the value of accreditation for the efficient delivery of public health services^6-8; however, because of the personnel and financial resources needed to complete the accreditation application, pursuing accreditation may not be efficient for all LHDs. To decide whether accreditation is an efficient use of scarce public health resources, LHDs should internally assess their capacity for the accreditation process to evaluate if they are prepared for the investment it will require.

Public health accreditation is a 7-step process that includes preapplication, application, document selection and submission, site visit, accreditation decision, reports, and reaccreditation. The first step, preapplication, includes a period during which the LHD self-assesses its capacity for the accreditation process. To assist LHDs with this task, PHAB created accreditation readiness checklists.⁹ Although these checklists help LHDs understand their readiness for the technical process of accreditation, they are limited in their assessment of broader leadership and the capacity of departmental culture.⁹ The National Association of County and City Health Officials (NACCHO) included accreditation questions in the Profile of Local Health Departments survey, but these questions are limited to intent to apply for accreditation.¹⁰ No tools are currently available with established reliability and validity that combine these elements to quantitatively measure an LHD’s capacity for accreditation. The availability of a tool to assess an LHD’s capacity for accreditation would inform the feasibility of initiating the time-intensive and costly accreditation process.

The objective of this study was to develop and score a reliable and valid survey that measures domains of technical and departmental capacity for accreditation at LHDs.

Methods

Survey Development

We generated domains of capacity for public health accreditation from the collection and review of PHAB checklists, NACCHO Profile survey questions, a review of the literature, and key informant interviews with accreditation experts and LHD leaders. We developed the survey through an iterative process that involved drafting questions, categorizing questions into domains, and having study coauthors review the questions from August through December 2014. The internal study team and key stakeholders (including LHD leaders and accreditation champions from across the United States) vetted the survey before creating a final version.

The 15-question final survey included 2 introductory questions to assess an LHD’s intent to apply for accreditation and intent to use PHAB standards: “Does your department intend to apply for accreditation?” and “Does your department plan to use PHAB standards to improve your work, regardless of your intent to formally seek accreditation?” We derived the question about using PHAB standards without intending to apply from interviews with LHD leaders, who indicated a desire to improve and align with standards but lacked the resources to formally apply for accreditation. The remaining 13 questions quantitatively assessed an LHD’s capacity for accreditation by using a 5-point Likert scale (where 1 = strongly disagree and 5 = strongly agree) to capture information on the support, structure, and steps the LHD has taken toward accreditation. Supplemental material associated with this article can be found with the online version of the article.

Study Population

We invited all LHDs in Colorado, Kansas, and Nebraska to participate. All 3 states have a decentralized state to local health agency governance relationship, meaning the LHDs are largely independent of the state health department. We emailed the electronic surveys to the directors of all 174 LHDs in the 3 states (54 LHDs in Colorado, 20 LHDs in Nebraska, and 100 LHDs in Kansas) from January through April 2015. In Kansas and Nebraska, we included our surveys in state data collection efforts that were required of LHDs. We sent our accreditation capacity survey and a Quality Improvement Maturity Survey (QIMS)^11,12 to all LHD directors in each state. We included the QIMS in addition to our accreditation capacity survey for the purposes of validation testing. We obtained responses to the QIMS, in addition to responses to our accreditation capacity survey, for the purpose of survey validation.

Descriptive Analysis

For the 2 introductory survey questions on accreditation intent and use of PHAB standards, we generated frequencies of each response. We calculated frequencies for the entire set of survey respondents and then separately for each state. We then calculated means, standard deviations, minimums, and maximums for the remaining 13 questions.

Psychometric Analysis

We analyzed and grouped survey responses by using a confirmatory maximum likelihood factor analysis with oblique rotations. We selected a confirmatory factor analysis rather than an exploratory factor analysis to allow us to test the theoretical framework established by PHAB in its accreditation checklists.¹³ This confirmatory approach argued for the existence of an established theoretical framework for accreditation capacity as evidenced through the PHAB checklists. This approach allowed us to confirm or reject that framework and associated latent constructs. We identified factor groupings with an eigenvalue ≥1, and we excluded factor groupings with an eigenvalue <1. We removed survey questions from the factor loadings if the communality, or the proportion of variation in that variable explained by the factors, was <0.1. We then scored responses to each survey item by multiplying the response (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree) by the weighted factor loading presented in Table 1. We calculated the factor score by adding the score for each item in the factor.

Table 1.

Factor loadings^a from a cross-sectional survey designed to measure local health departments’ capacity for accreditation, United States, 2015^b

Question	Factor 1	Factor 2	Factor 3	Factor 4c	Communalityd
The Board of Health or other governing entity supports the health department’s seeking PHAB accreditation.	1.04	–0.06	–0.04	–0.05	1.00
The director of the health department supports the health department’s seeking PHAB accreditation.	0.41	0.35	–0.05	–0.10	0.44
The health department has completed an initial self-study or review of the standards, measures, and required documentation to determine areas of strength and opportunities for improvement.	0.08	0.79	0.00	0.19	0.74
The health department has the capability to produce electronic documentation.	–0.17	0.79	–0.09	–0.30	0.53
The health department has developed and begun implementation of plans to address identified opportunities for improvement.	0.10	0.50	0.15	0.10	0.45
The health department has evidence that collaboration with tribal, state, or community partners and stakeholders is inherent in how the health department conducts planning, develops policy, and completes its work.	–0.05	–0.08	1.06	–0.04	1.00
The health department has a community health improvement plan that meets PHAB accreditation standards.	0.04	0.27	0.28	–0.12	0.29
The health department has a process to systematically review department policies and procedures and revise and date them, as needed.	0.12	0.21	0.26	0.12	0.25
The health department has sought and secured technical assistance to address opportunities for improvement.	–0.12	0.11	–0.07	0.55	0.31
The appointing authority for the health department director supports the health department’s seeking PHAB accreditation.e	–0.11	0.10	–0.01	–0.05	0.01
The health department has a community health assessment that meets PHAB accreditation standards.e	–0.06	0.10	0.08	0.02	0.02
The health department has a department strategic plan that meets PHAB accreditation standards.e	0.03	–0.06	–0.03	–0.02	0.01
The health department has the capability to establish an internal electronic filing system, with a separate file for each measure, to be a repository for PHAB documentation.e	–0.01	–0.05	0.14	–0.06	0.02
Eigenvaluef	2.10	2.90	2.30	0.50

Abbreviation: PHAB, Public Health Accreditation Board.

^a Factor loadings indicate how variables were weighted for each factor and the correlation between the variables and the factor.

^b Survey responses from 153 local health departments were analyzed. For each question, respondents ranked their health departments from 1 (strongly disagree) to 5 (strongly agree). The survey questions loaded to 4 factors.

^c Excluded because the eigenvalue was <1.0.

^d Communality measures the proportion of variation in that variable explained by the factors.

^e Excluded because communality was <0.1.

^f An eigenvalue is the amount of variable variation explained by the factor.

We assessed survey reliability by calculating the Cronbach α for each domain.^14,15 We assessed validity by using both predictive validity and construct validity.¹⁵ We assessed the predictive validity of each factor that loaded in our psychometric analysis by comparing each respondent’s factor score with the responses from an already established instrument, including the NACCHO profile¹⁰ and QIMS.^11,12 First, we compared the factor score for the support for accreditation domain with the sum of responses to 2 questions (each question response ranged from 1 = strongly disagree to 5 = strongly agree) related to quality improvement staff member participation from the QIMS. We compared the factor score for the preparation domain with responses from the 2013 NACCHO Profile related to accreditation intent. We compared the factor score for the third domain, planning and approach, with the sum of responses to 2 questions (each question response ranged from 1 = strongly disagree to 5 = agree) related to quality improvement capacity from the QIMS. We also conducted a construct validity test by comparing factor scores from the accreditation capacity survey at the state level with state investments in quality improvement and accreditation. We tested the theory that we would expect higher accreditation capacity to be an outcome of higher system-level investments in quality improvement and accreditation. Recent work shows that of the 3 included states, Nebraska received the highest investment in quality improvement. Kansas had fewer system-level investments in quality improvement than Nebraska but more than Colorado. Therefore, the states with the highest to lowest amount of system-level investments were Nebraska, Kansas, and Colorado.

We performed all statistical analyses by using Stata IC version 13.1.¹⁶ The Colorado Institutional Review Board and the University of Nebraska Institutional Review Board approved all research methods and informed consent procedures.

Results

Of 174 surveys distributed to LHD directors, 153 (88%) were completed. All LHD directors in Kansas (n = 100) and Nebraska (n = 20) responded, and 33 of 54 (61%) LHDs in Colorado responded.

Of 174 respondents to the question on intention to apply for accreditation, 32 (18%) reported an intention to apply for accreditation in the next 2 years, 71 (41%) reported an intention to apply for accreditation in >2 years, and 71 (41%) reported no intention to apply for accreditation. Responses varied by state. Most of the responding LHDs in Colorado (n = 17, 53%) reported no intention to apply for accreditation, whereas in Kansas, 44 (44%) LHDs reported no intention to apply for accreditation, 49 (49%) LHDs reported an intention to apply for accreditation in >2 years, and 7 (7%) LHDs reported an intention to apply for accreditation in the next 2 years. Conversely, in Nebraska, only 2 (10%) LHDs reported no intention to apply for accreditation, 7 (35%) reported an intention to apply for accreditation in >2 years, and 11 (55%) intended to apply for accreditation in the next 2 years.

In response to the second question, on the intention to use PHAB standards, 26 (15%) respondents said they had not considered using PHAB standards, 45 (26%) respondents had considered using PHAB standards but had not decided to use the standards, 79 (45%) respondents planned on using PHAB standards, and 24 (14%) respondents were already using PHAB standards. In Nebraska, all respondents reported either planning to use PHAB standards or currently using PHAB standards despite no state requirement for them to do so.

In response to statements assessing accreditation capacity, mean scores ranged from 2.9 for “The health department has completed an initial self-study or review of the standards, measures, and required documentation to determine areas of strength and opportunities for improvement” to 3.9 for “The health department has a community health assessment that meets PHAB accreditation standards” (Table 2).

Table 2.

Descriptive findings from a cross-sectional survey designed to measure local health departments’ capacity for accreditation, United States, 2015^a

Question	Mean (SD)
The health department has a community health assessment that meets PHAB accreditation standards.	3.9 (1.0)
The director of the health department supports the health department’s seeking PHAB accreditation.	3.7 (1.1)
The health department has evidence that collaboration with tribal, state, or community partners and stakeholders is inherent in how the health department conducts planning, develops policy, and completes its work.	3.7 (0.9)
The health department has a community health improvement plan that meets PHAB accreditation standards.	3.6 (1.2)
The health department has the capability to produce electronic documentation.	3.6 (1.1)
The health department has a process to systematically review department policies and procedures and revise and date them, as needed.	3.5 (1.0)
The appointing authority for the health department director supports the health department’s seeking PHAB accreditation.	3.4 (0.9)
The Board of Health or other governing entity supports the health department’s seeking PHAB accreditation.	3.3 (0.9)
The health department has the capability to establish an internal electronic filing system, with a separate file for each measure, to be a repository for PHAB documentation.	3.3 (1.2)
The health department has developed and begun implementation of plans to address identified opportunities for improvement.	3.2 (1.0)
The health department has sought and secured technical assistance to address opportunities for improvement.	3.0 (1.1)
The health department has a department strategic plan that meets PHAB accreditation standards.	3.0 (1.1)
The health department has completed an initial self-study or review of the standards, measures, and required documentation to determine areas of strength and opportunities for improvement.	2.9 (1.1)

Abbreviation: PHAB, Public Health Accreditation Board.

^a Survey responses from 153 local health departments were analyzed. For each question, the respondent ranked its health department from 1 (strongly disagree) to 5 (strongly agree).

Psychometric Analysis

The factor analysis identified 3 factors that had an eigenvalue ≥1 (Table 1). Two survey items loaded on the first factor: “The director of the health department supports the health department’s seeking PHAB accreditation” and “The Board of Health or other governing entity supports the health department’s seeking PHAB accreditation.” We designated this factor as “support for accreditation” because both questions related to perceived support from authoritative entities.

Three survey items loaded on the second factor: “The health department has the capability to produce electronic documentation,” “The health department has completed an initial self-study or review of the standards, measures, and required documentation to determine areas of strength and opportunities for improvement,” and “The health department has developed and begun implementation of plans to address identified opportunities for improvement.” We designated this factor as “preparation” because all 3 items were related to elements that are typically completed before the application process.

Three survey items loaded on the third factor: “The health department has a community health improvement plan that meets PHAB accreditation standards,” “The health department has a process to systematically review health department policies and procedures and revise and date them, as needed,” and “The health department has evidence that collaboration with tribal, state, or community partners and stakeholders is inherent in how the health department conducts planning, develops policy, and completes its work.” We designated this factor as “planning and approach” because all 3 items related to elements associated with accreditation standards.

We identified a fourth factor but excluded it because it had an eigenvalue of 0.5. One item loaded on this factor and was thus removed: “The health department has sought and secured technical assistance to address opportunities for improvement.” The factor analysis also suggested removing 4 survey items because they had a commonality <0.1.

After multiplying the factor loadings (Table 1) by the lower (1 = strongly disagree) and upper (5 = strongly agree) bound for survey responses, the total factor score for factor 1 (ie, support for accreditation) ranged from 1.5-7.3. The total score for factor 2 (ie, preparation) ranged from 2.1-10.4. The total score for factor 3 (ie, planning and approach) ranged from 1.6-8.0. Higher scores represented higher accreditation capacity (Box).

Box.

Instructions for scoring a survey designed to measure local health departments’ capacity for Public Health Accreditation Board (PHAB) accreditation, United States, 2015

There are 3 domains of accreditation readiness: (1) support for accreditation, (2) preparation, and (3) planning and approach.

Support for Accreditation Score

The Support for Accreditation score was calculated by multiplying the responses for the following 2 questions by their factor score in Table 1. The 2 weighted scores were then added.

The director of the health department supports the health department’s seeking PHAB accreditation.

1. Strongly agree = 5 × 0.4134

2. Agree = 4 × 0.4134

3. Neutral = 3 × 0.4134

4. Disagree = 2 × 0.4134

5. Strongly disagree = 1 × 0.4134

The Board of Health or other governing entity supports the health department’s seeking PHAB accreditation.

1. Strongly agree = 5 × 1.0403

2. Agree = 4 × 1.0403

3. Neutral = 3 × 1.0403

4. Disagree = 2 × 1.0403

5. Strongly disagree = 1 × 1.0403

Preparation Score

The Preparation score was calculated by multiplying the responses for the following 3 questions by their factor score in Table 1. The 3 weighted scores were then added.

The health department has the capability to produce electronic documentation.

1. Strongly agree = 5 × 0.7862

2. Agree = 4 × 0.7862

3. Neutral = 3 ×× 0.7862

4. Disagree = 2 × 0.7862

5. Strongly disagree = 1 × 0.7862

The health department has completed an initial self-study or review of the standards, measures, and required documentation to determine areas of strength and opportunities for improvement.

1. Strongly agree = 5 × 0.7949

2. Agree = 4 × 0.7949

3. Neutral = 3 × 0.7949

4. Disagree = 2 × 0.7949

5. Strongly disagree = 1 × 0.7949

The health department has developed and begun implementation of plans to address identified opportunities for improvement.

1. Strongly agree = 5 × 0.4996

2. Agree = 4 × 0.4996

3. Neutral = 3 × 0.4996

4. Disagree = 2 × 0.4996

5. Strongly disagree = 1 × 0.4996

Planning and Approach Score

The Planning and Approach score was calculated by multiplying the responses for the following 3 questions by their factor score in Table 1. The 3 weighted scores were then added.

The health department has a community health improvement plan that meets PHAB accreditation standards.

1. Strongly agree = 5 × 0.2768

2. Agree = 4 × 0.2768

3. Neutral = 3 × 0.2768

4. Disagree = 2 × 0.2768

5. Strongly disagree = 1 × 0.2768

The health department has a process to systematically review department policies and procedures and revise and date them, as needed.

1. Strongly agree = 5 × 0.2574

2. Agree = 4 × 0.2574

3. Neutral = 3 × 0.2574

4. Disagree = 2 × 0.2574

5. Strongly disagree = 1 × 0.2574

The health department has evidence that collaboration with tribal, state, or community partners and stakeholders is inherent in how the health department conducts planning, develops policy, and completes its work.

1. Strongly agree = 5 × 1.0614

2. Agree = 4 × 1.0614

3. Neutral = 3 × 1.0614

4. Disagree = 2 × 1.0614

5. Strongly disagree = 1 × 1.0614

Reliability and Validity

The Cronbach α was .734 for the support for accreditation domain, .723 for the preparation domain, and .582 for the planning and approach domain. The correlation coefficient for the support for accreditation domain was 0.21. The correlation coefficient for the preparation domain was 0.57. The correlation coefficient for the planning and approach domain was 0.50. This resulted in an average factor correlation of 0.43.

The factor scores for each state generated through this analysis aligned with the rank ordering of system-level investments (Table 3). For the support for accreditation domain, Nebraska had the highest score (5.8), followed by Kansas (4.8) and Colorado (4.1). For the preparation domain, Nebraska had the highest score (8.3), followed by Kansas (6.5) and Colorado (6.1). For the planning and approach domain, Nebraska had the highest score (6.1), followed by Colorado (5.8) and Kansas (5.7).

Table 3.

Public Health Accreditation Board accreditation capacity scores calculated for states participating in a cross-sectional survey of local health departments, United States, 2015

State	Support for Accreditation, Mean (SD)a	Preparation, Mean (SD)b	Planning and Approach, Mean (SD)c
Colorado (n = 33)	4.1 (2.4)	6.1 (2.1)	5.8 (1.4)
Kansas (n = 20)	4.8 (1.1)	6.5 (1.7)	5.7 (1.3)
Nebraska (n = 100)	5.8 (1.4)	8.3 (1.7)	6.1 (1.6)

Abbreviation: SD, standard deviation.

^a Two survey questions, each measured on a scale from 1 (strongly disagree) to 5 (strongly agree), loaded on the “support for accreditation” domain. Scores from each question were multiplied by the factor loadings from Table 1 and added together; thus, the range for this domain was 1.5-7.3.

^b Three survey questions, each measured on a scale from 1 (strongly disagree) to 5 (strongly agree), loaded on the “preparation” domain. Scores from each question were multiplied by the factor loadings from Table 1 and added together; thus, the range for this domain was 2.1-10.4.

^c Three survey questions, each measured on a scale from 1 (strongly disagree) to 5 (strongly agree), loaded on the “planning and approach” domain. Scores from each question were multiplied by the factor loadings from Table 1 and added together; thus, the range for this domain was 1.6-8.0.

Practice Implications

Encouraging LHDs to pursue accreditation is a national priority, with the goal of promoting quality and performance of LHDs to improve, promote, and protect health. This study developed and scored a 15-question survey instrument that quantitatively assessed capacity for accreditation. We found that accreditation capacity for LHDs depended on 3 latent constructs: support for accreditation capacity (2 items), preparation (3 items), and planning and approach (3 items). Although only 8 of the 13 quantitative questions factored into the accreditation capacity score, the remaining 5 quantitative questions may still be important conceptual attributes of accreditation capacity. LHDs can use this survey instrument to internally assess their own accreditation capacity as part of the preapplication step of the accreditation process. System-level partners may also use the survey instrument to assess a group of LHDs to understand how best to help them prepare for accreditation.

Limitations

This study had several limitations. First, the data were based on a self-reported, opinion-based survey. As such, the data could not be interpreted as an accreditation readiness measurement tool because there could be important factors that the survey did not capture, and the scores from the tool were not triangulated with confirmation of accreditation. Despite this limitation, an LHD could use this tool to internally assess its capacity to pursue accreditation. Second, we used self-reported responses from LHDs in only 3 states to generate the factor loadings; therefore, the results may not be generalizable to other states. We may not have been able to identify state-level factors, such as the state’s commitment to the accreditation process, which might have influenced the results. Third, not all LHDs from Colorado responded to the survey, thereby creating the possibility of response bias because each LHD in that state decided whether to participate. To assess response bias, we performed χ² tests of independence to determine if response status was significantly associated with accreditation intent (as reported to NACCHO), LHD jurisdiction size, and longevity of public health directors. We found no significant association between response status and these variables; however, our sample did have a high proportion of agencies with large jurisdiction sizes. We concluded that response bias in Colorado was nonexistent to minimal based on the variables we evaluated. We assume that all LHDs in Kansas and Nebraska responded because we included our surveys in state data collection efforts that were required of LHDs.

Fourth, the predictive validity assessment was limited because of the limited availability of other data from validated surveys collected from our respondents. Fifth, each state had its own accreditation and quality improvement landscape, with various system-level investments between states. Future research should investigate the impact of system-level investments and initiatives for quality improvement on accreditation capacity at the LHD level. It is also worth noting that the public health accreditation landscape is changing rapidly; therefore, the descriptive results related to the number of LHDs that were accredited at the time of survey completion have likely changed. However, we do not anticipate a different relationship from the psychometric analysis. Sixth, this study does not suggest a certain accreditation capacity cutoff score at which an LHD should consider itself ready to pursue accreditation. However, it does present the lower bounds and upper bounds for each domain for the LHD to place their score on a spectrum. Future research should investigate the association between the accreditation capacity score and ease of accreditation application and success, which would help determine whether our survey is better suited for LHDs with certain characteristics. Future research involving this survey of accreditation capacity should examine the extent to which the included questions align with organizational theories and the extent to which the survey responses differ based on type of respondent (eg, LHD director vs performance improvement manager).

Finally, these data were collected in 2015, and overall support and acceptance of accreditation continues to increase, as evidenced by the tripling of LHDs engaged in accreditation from 2013 to 2018 (from 6% to 21%). In Nebraska, no LHDs were accredited during the data collection period, whereas 5 LHDs and the state health department were accredited as of June 2019. In Colorado, 1 LHD was accredited in 2015, whereas 6 LHDs and the state health department were accredited as of June 2019. In Kansas, 3 LHDs were accredited during the data collection period, whereas the Kansas state health department and 1 other LHD were accredited as of June 2019.

Conclusion

LHDs can use the survey developed and tested in this analysis during the preapplication step of the accreditation process. Survey results can quantitatively inform LHDs about their capacity to pursue the accreditation process. Because this work found accreditation to be a multidimensional attribute, LHDs should focus on each dimension of accreditation capacity, and address barriers to each, when preparing for the time-intensive and costly accreditation process.