Reliability and Validity of the Assessment for Disaster Engagement with Partners Tool (ADEPT) for Local Health Departments

Abstract

Objective

This study presents reliability and validity findings for the Assessment for Disaster Engagement with Partners Tool (ADEPT), an instrument that can be used to monitor the frequency and nature of collaborative activities between local health departments (LHDs) and community-based organizations (CBOs) and faith-based organizations (FBOs) for disaster preparedness, response, and recovery.

Methods

We used formative research to develop the instrument by ranking LHDs according to their disaster outreach and engagement activities. We validated the scale through a 2011 national survey of disaster preparedness coordinators (n=273) working in LHDs. We reduced the original measure of 25 items to a final measure comprising 15 items with four dimensions: (1) communication outreach and coordination, (2) resource mobilization, (3) organizational capacity building, and (4) partnership development and maintenance. We used internal consistency reliability m correlation and factor analysis to validate the measure.

Results

Using internal consistency reliability, we found reasonable inter-item reliability for the four hypothesized dimensions (Cronbach's alpha: 0.71–0.88). These four dimensions were confirmed through correlation and factor analysis (Varimax rotation).

Conclusion

Higher scores on all four dimensions of ADEPT for organizational respondents suggest that more activities were conducted for inter-organizational preparedness in those organizations than in organizations whose respondents had lower scores. This finding implies that organizations with higher ADEPT scores have more active relationships with CBOs/FBOs in the realm of preparedness, a key element for creating community resilience for emergencies and disaster preparedness.

It has been well documented that before, during, and after disasters, vulnerable populations frequently turn first to trusted community-based organizations (CBOs) and faith-based organizations (FBOs) for disaster information, resources, shelter, or other assistance rather than to government, media, or official sources.^1–9 For example, CBOs and FBOs provided essential services to victims and residents following the 2001 World Trade Center attack.^10–13 Recent events, such as the earthquake in Haiti, Hurricanes Irene and Sandy on the East Coast of the United States, and serious tornadoes and wildfires, suggest the importance of CBOs/FBOs for reaching vulnerable populations before, during, and after disasters, with some criticizing ineffective governmental response following Hurricane Katrina compared with CBOs/FBOs^6,7,9,14 that often serve vulnerable populations.^8,15 As local health departments (LHDs) have become more engaged in disaster response, priorities include establishing inter-organizational networks^16–20 with CBOs and FBOs, as they are integral partners for building resilient communities.^9,20,21 Yet, LHDs may not always value these linkages or even be able to sustain them due to limited resources or mandates.^20–22

Despite calls for cultivating inter-organizational relationships with CBOs and FBOs on the part of LHDs for disaster preparedness and community resilience,^22–26 only a few empirical studies describe the pragmatics of fostering such linkages.^{21,22,27–34} In a study of California LHDs, Lurie et al. found that partnerships with CBOs were often not the norm in disaster and bioterrorism response.²¹ Mays et al. noted that while more than 75% of LHDs reported having communication networks with CBOs, the media, and the public, only 46% of LHDs dedicated time to maintaining these networks, and only 24% felt their efforts were useful.²² Several studies also have examined the link between the structure and strength of inter-organizational social network ties and effective disaster response.^29,30

Moreover, many of the same obstacles to and facilitators for organizational linkages in other areas of public health may apply here. Lerner and colleagues examined a convenience sample of seven “model communities,” defined as communities that have good working relationships among public health and emergency services agencies, such as fire department, law enforcement, and emergency medical services. Important elements for building linkages included holding regularly scheduled meetings; developing response plans together that meet unique local circumstances; working together daily on disaster- and non-disaster-related activities; and having a strong leader driving the collaboration, shared resources, and leveraged funding to accomplish goals.³¹ Another relevant study of linkages between health centers and community emergency preparedness and response planning initiatives identified several health center-level demographic and experiential factors associated with linkages, including rural location, prior disaster experiences, and high perceived risk for disasters. Barriers to linkages in this study included staff limitations and time restraints, low funding, the potential role of the health center not being understood by emergency planners, and a lack of strong leadership.³² These findings have been confirmed by a more recent study conducted in Los Angeles³³ as well as several national studies.^34,35

We addressed the issue of what LHDs do with their organizational partners to have functional ties for disaster preparedness and support, with an emphasis on frequency of activities rather than strength of relationships. We named the project the Assessment for Disaster Engagement with Partners Tool (ADEPT) and developed scales based on extensive formative research as well as organizational theory, suggesting that inter-organizational coordination meets needs that individual organizations cannot meet alone.^36–38 Coordination involves integrating, linking, or otherwise bringing together various aspects of an organization (e.g., people and departments) or a group of organizations for information exchange and/or resources to accomplish common goals associated with beneficial outcomes,³⁶ either through formal coordination programming^39,40 or less formal feedback coordination.^39,41

ADEPT was also informed by emerging ideas about public health systems as proposed by the Institute of Medicine,¹⁸ which considers LHDs to be part of a larger public health system. These ideas also resonate with emerging directives from the federal government that are now reframing disaster preparedness and response as community disaster resilience.^9,16–20,42 A central facet of this approach is mobilizing or engaging communities and their organizations and leaders in partnerships with nongovernmental agencies prior to disasters to assure coordinated and broad-reaching approaches in the face of disasters. Finally, ADEPT items also correspond to disaster preparedness capabilities for health departments as articulated by the Centers for Disease Control and Prevention, specifically capability 1 (i.e., community preparedness) and functions 2, 3, and 4, all of which entail community engagement.¹⁹

METHODS

Formative research

After an extensive literature review, we conducted a modified environmental scan of websites of organizations (i.e., CBOs and FBOs) that are active in disasters. We used lists of LHDs that are members of the National Association of County and City Health Officials (-NACCHO) to generate samples for our formative research. The purposive sample of key informants from our NACCHO and CBO/FBO lists included representatives from a variety of organizational types, sizes, missions, levels and types of activity, and jurisdictional characteristics. We then conducted qualitative, semi-structured interviews to finalize the conceptual development of survey items that comprise the multidimensional ADEPT scales for inter-organizational community engagement. We conducted one set of qualitative interviews with local and regional health department disaster coordinators (n=13) and a second set of interviews with staff of local and regional CBOs and FBOs (n=20) for whom disaster and emergency work was part of their mission.³⁵ We conducted key informant interviews by telephone using a recruitment and data collection process approved by the University of California, Los Angeles (UCLA), Institutional Review Board.

Findings from key informant interviews provided insight into the type of organizational, training, and staff and resource constraints and facilitators LHDs and CBOs/FBOs face when attempting to create and sustain ongoing partnerships for disasters. Additionally, based on both real and hypothetical scenarios regarding disasters described during these interviews, we elicited information about the types of activities these organizations were conducting that enabled them to have successful inter-organizational partnerships in support of disaster preparedness and response.³⁵

Based on an analysis of key informant interviews, supporting literature, an environmental scan, and the extensive organizational experience and input of both the investigative team and our Advisory Board, we came up with scale items and dimensions in an iterative process. The Advisory Board members included 12 health professionals and leaders from academic institutions and public health organizations who have worked in communities in the area of disaster preparedness and response and, more broadly, community development. The four domains or dimensions of activities encompassed communication outreach, resource mobilization, organizational capacity building for disasters, and partnership development and maintenance. Following is a brief description of each domain.

Communication outreach and disaster coordination.

This domain encompasses a very common set of activities carried out with CBOs and FBOs by LHDs for disaster preparedness and response, comprising the creation, distribution, and promotion of disaster information materials and resources. Whether through in-person sessions, community events, print materials, online methods, organized campaigns, or warning systems, working with CBOs and FBOs is one way to distribute or amplify many forms of disaster communication.^43,44

Resource mobilization for disasters.

Often, health departments or other governmental agencies do not have the space, personnel, or goods/services to distribute to people, especially those in hard-to-reach communities, before, during, and after disasters. A number of studies, as well as our own findings, suggest that CBOs and FBOs often serve as organizational extenders of services, space, volunteers, and resources for communities in disasters.^{2,13,15,27,28,45}

Organizational capacity building for disasters.

This set of activities resonates with what some call business continuity planning for disasters. It suggests that as participants in creating resilient communities, CBOs and FBOs must themselves become disaster prepared—creating plans, gaining competencies, stocking supplies, and training personnel and members—so that in the event of a disaster, those assets can be used to help members, clients, or other constituents.^{20,27,28,45,46}

Partnership development and maintenance for disasters.

A final set of inter-organizational activities integral to the process of engaging with local organizations in disasters is linked to general activities that promote and sustain meaningful inter-organizational linkages or partnerships with a disaster focus. These linkages are key in community-based planning to develop and enact community-wide preparedness and response plans, participate in community drills, and ensure ongoing coordination and communication during an event.^{20,29–32,45,46}

Domain item development

Using the four domains that emerged from our formative research, we developed a questionnaire to assess the frequency with which LHDs undertake activities in each domain in collaboration with CBOs/FBOs, referred to as the ADEPT scale. Item domains are seen to cut across characteristics of LHDs regardless of jurisdiction type and size. Some of the items were created de novo based on our formative research, and some were adapted from literature review findings of existing partnership and collaboration development materials. Generally, we attempted to create items that reflected the most common types of activities that might be enacted within these “generic” domains of inter-organizational engagement, defined as the types of things that all agencies do for partnership development. Survey items were also subject to external review by Advisory Board members and subject-matter experts.

Psychometric and statistical analysis

To describe the validity and reliability of ADEPT scale domains, we conducted a national survey of disaster preparedness coordinators at LHDs (n=273) from August through December 2011. Using the NACCHO database of 2,864 LHDs, we applied a probability-proportional-to-size sampling design to generate a stratified random sample of 750 LHDs that reflects the national distribution of large (>250,000), medium (25,000–250,000), and small (<25,000) populations.⁴⁷

Survey data were collected from disaster preparedness coordinators at LHDs using an online data collection system. All respondents filled out UCLA Institutional Review Board informed consent forms. The survey included the ADEPT items as well as items concerning organizational, jurisdictional, and personal characteristics. Survey completion took an average of 30 minutes. Initially, there were 25 Likert scale items for the four conceptualized domains. To create a more efficient instrument, we pared items to 15 without sacrificing internal consistency, item discriminability, or variance accounted for in the factor analysis.

We analyzed all of our items using Exploratory Principal Components Factor Analysis with Varimax rotation. We performed the analysis multiple times, varying the number of factors fitted to investigate the optimal number of factors to fit. We considered the following when determining the optimal number of factors: total proportion of the variance accounted for, factor eigenvalues, scree plot (for graphing the eigenvalue against the factor number, giving an idea of which factors account for the variance explained), factor loading patterns, item clustering patterns, and face validity. We used classical test theory approaches to evaluate the psychometric properties of the multidimensional measure.^48,49

We also assessed measures for internal consistency reliability or the pattern of inter-item correlations within each factor using Cronbach's alpha.⁵⁰ We measured an item's internal consistency by the average inter-item correlation (i.e., the average pairwise correlation of an item with all other items). We measured an item's discriminability by the item total correlation (i.e., the item's correlation with the sum of all other items). We evaluated item reliability by measuring the difference between the factor's reliability and the factor's reliability with the item removed. Items presenting substantial consistency or discriminability differences, an increase in reliability when removed, or diminished factor loading values across all factors were identified as poorly performing items and were removed. We performed all statistical analyses using SAS^® version 9.2.⁵¹

RESULTS

In the study sample of disaster preparedness coordinators interviewed (n=273), the jurisdictional size of health departments reflected national distributions, with approximately 18.5% of respondents from small LHDs (<25,000 people served) and 13.3% of respondents from larger LHDs (>250,000 people served) (data not shown). Despite a low response rate (42%), due in part to lack of information about disaster coordinators in LHDs—particularly of medium-sized populations—our final study sample did reflect the distribution jurisdiction size, with most of the sample from medium or large jurisdictions, as described elsewhere.⁴⁷ Approximately 54% of those surveyed worked more than half-time in emergency and disaster preparedness, and 58% of respondents had worked in emergency and disaster preparedness for >6 years. Women comprised 67% of the sample, and 55% of respondents were 45–65 years of age (data not shown).

Prior to factor analysis, we also created Pearson's correlations on all 25 items and found that correlations had moderate to good strength and also suggested dimensional structure we had posited (Figure).

Figure.

Inter-item correlation matrix for ADEPT questionnaire items among a national sample of LHD disaster preparedness coordinators (n=273): Los Angeles, California, 2011^a

^aNumbers represent original survey item numbers.

ADEPT = Assessment for Disaster Engagement with Partners Tool

LHD = local health department

Psychometric measurements for the 15-item scale showed a robust Cronbach's alpha (0.89) across all items (Table 1). For the four domains conceptualized in the reduced 15-item scale, the alphas ranged from 0.71 to 0.81. Domains with the strongest alphas were communication outreach and coordination (alpha = 0.81) and partnership development and maintenance (alpha = 0.80). Resource mobilization, the domain with the lowest alpha (0.71), was possibly the result of more variation in the types of resources considered when responding to items for this type of activity. Both item consistency and discriminability measures suggest that the items within each domain or factor are measuring the same underlying construct. All items exhibited a moderate degree of consistency (average inter-item correlations ranging from 0.41 to 0.61) and discriminability (item total correlations ranging from 0.50 to 0.69) (data not shown).

Table 1.

Internal consistency reliability of ADEPT domains measured among a national sample of LHD disaster preparedness coordinators (Cronbach's alpha = 0.89): Los Angeles, California, 2011

^a25-item scale

^b15-item scale

ADEPT = Assessment for Disaster Engagement with Partners Tool

LHD = local health department

SD = standard deviation

Q = question

CBO = community-based organization

FBO = faith-based organization

EPDR = emergency preparedness and disaster response

As shown in Table 2, factor analysis results suggest that the data are parsimoniously described by the four-factor model based on factor loading and item clustering patterns. For the four-factors solution, 62.5% of the variance was accounted for, and the rotated factor matrix (Varimax) showed all factor eigenvalues to be >1 with three factors >2. Here, as in the results for internal consistency reliability, the highest factor scores and the most variance accounted for were for partnership development/maintenance (19.6%) and communication outreach/coordination (15.0%). For organizational capacity building, 14.9% of the variance was accounted for, while only 13.0% of the variance was accounted for in the resource mobilization domain.

Table 2.

Exploratory principal components factor analysis results and factor scores for ADEPT among a national sample of LHD disaster preparedness coordinators (n=273): Los Angeles, California, 2011^a

^aBoxes indicate items with similar factor scores.

ADEPT = Assessment for Disaster Engagement with Partners Tool

LHD = local health department

Q = question

EPDR = emergency preparedness and disaster response

CBO = community-based organization

FBO = faith-based organization

DISCUSSION

This study capitalized on calls to operationalize public health systems, with particular emphasis on the interface between LHDs and CBOs/FBOs.¹⁸ We based our ADEPT scale on the frequency of activities that LHD disaster preparedness personnel say they conduct to achieve functional linkages with CBOs and FBOs for emergency preparedness and disaster response. Items could also be conceptualized as performance indicators, as they are conducted as part of a disaster coordinator's professional role.

Numerous studies have made a strong case for the importance of inter-organizational linkages to increase community disaster resilience.^28–34 However, studies reviewed often present the activities undertaken by organizations in support of disaster preparedness as disparate activities rather than interrelated sets of activities, as we have attempted to do in this study.

In the development of our ADEPT scale, we followed classical measurement theory, which promotes the ideas of conceptual themes or domains of action and cognition, which are then operationalized and measured. In that schema, operationalized items refer to a subset or sample of all the potential items that could be used to measure that construct.⁵⁰ By organizing activities that support community disaster resilience in this manner, we devised a valid psychometric scale so that collection of this information can be replicated, standardized, and better documented across organizations and contexts.

It should also be noted that we collected survey data using a 25-item scale that had high internal consistency (Cronbach's alpha = 0.92) and factor analysis results that also supported the same four-factor model. The impetus behind reducing the number of items from 25 to 15 was mainly due to the feasibility of its implementation rather than statistical outcomes, as a 25-item scale is unwieldy, especially in practice-based contexts. However, the 25-item scale can still be used in research.

Limitations and strengths

This study was subject to several limitations. One limitation was a less-than-optimal survey response rate; however, the response rate had a minimal impact on the validity of study findings as regards instrument development. Another limitation was the frame of measure. We asked about the frequency of activities performed from the perspective of the LHD disaster preparedness coordinator, but the question could also have been asked from the perspective of a CBO/FBO staff person, health or emergency organization, school, or business, and there could be more emphasis on response rather than preparedness. Finally, we did not include many explicit questions about disaster drills and exercises. Each of the four types of activities we outlined could also be important components of community-based drills and exercises.

A strength of our multidimensional ADEPT measure was the compatibility between the content of the items and the context of their use. There is nothing as practical as a good theory, and the items in our measure push the field toward considering underlying theories that support notions of organizational behaviors and competencies or capabilities regarding disaster preparedness and response.¹⁹ While social networking tools are available to assess the strength of inter-organizational linkages for disasters,^20,30 fewer studies have assessed what is done in support of those linkages or how those linkages and associated activities potentially change disaster outcomes.

CONCLUSIONS

Development of this type of assessment tool can allow LHDs and especially their disaster coordinators to systematically assess how important their participation in these sets of activities is for their organizational mission and identify deficits. Moreover, if scale items can also be understood as a measure of organizational performance, they may also be used in evaluation models that attempt to assess outcomes.⁵² Participation by an LHD in specific ADEPT activity sets may be used to improve efforts for disaster readiness and response. These dimensions can also be incorporated into drills and exercises. Finally, further research could test whether or not these indicators predict community disaster recovery or identify those factors that cause some communities to be more resilient in disasters.