On the Validity of Validation: A Commentary on Rufat, Tate, Emrich, and Antolini’s “How Valid Are Social Vulnerability Models?”

In “How Valid Are Social Vulnerability Models?” (Rufat et al. 2019), the authors compared social vulnerability models using Hurricane Sandy land-fall event outcomes as their evaluation example. Using spatial regression, they compared four vulnerability models across four Federal Emergency Management Agency (FEMA) outcomes to determine the empirical validity of each model. The FEMA outcomes include assistance applicants, affected renters, housing flood damage, and property loss. Although the article presents a statistically complex and seemingly open-and-shut case for using the authors’ social vulnerability profiles method in examining disaster outcomes, the argument includes several misunderstandings and weaknesses, especially regarding the intent and applicability of the Centers for Disease Control and Prevention Social Vulnerability Index and database (CDC SVI). An outline of our concerns and a brief description of our own ongoing validation efforts follow an overview of the CDC SVI.

CDC SVI Meets a Critical Need

The CDC SVI was created pursuant to the Pandemic and All-Hazards Preparedness Act of 2006, which cited public health and medical preparedness and response capabilities as a critical need for the nation (CDC SVI 2020). To comply, the Geospatial Research, Analysis, and Services Program (GRASP) at the CDC/Agency for Toxic Substances and Disease Registry (ATSDR) created a database, index, and mapping application to assist state, local, and tribal disaster management officials with identifying locations of their most socially vulnerable populations before, during, and after a disaster. GRASP developed the CDC SVI with an emphasis on flexibility, recognizing the random uncertainty inherent in hazard events, including type of disaster, time of day, season of the year, and social and physical geography.

Social vulnerability itself is a human construct. There is no essential inherent human condition that identifies an individual or community as socially vulnerable. Thus, we approximate vulnerability by measuring other societal attributes. Social science researchers at GRASP selected attributes viewed as contributing most to social vulnerability in the context of hazard reduction after examining the considerable sociological and natural hazards literatures (Flanagan et al. 2011). The attributes are operationalized as population-level census variables, with the census being the optimal data source because it is collected on a regular basis at the tract level nationwide.

Except for the 2010 CDC SVI, when the U.S. Census Bureau did not collect disability data, each CDC SVI tool (years 2000, 2014, 2016, and 2018) comprises fifteen census variables. Each variable is ranked from highest to lowest vulnerability across all census tracts in the nation (and separately, in the individual states) with a nonzero population. The mapping of these data reveals geographic patterns of potential vulnerability to hazards to be used in all phases of the disaster cycle—preparedness, response, recovery, and mitigation (Morrow 1999).

The CDC SVI tool assists public health officials with preparing for, and responding to, emergency meteorological and geological events, disease outbreaks, and human-caused incidents. The CDC SVI is national in geographic scope, universal in its applicability to hazard types, and designed to be used without regard to the skill set or resources of its users. The database is open to variable weighting of indicators, the omission or addition of individual indicators, the omission of tangential themes, or the importation of external data sets. Moreover, percentages and raw numbers are included in the database to enable responders to better allocate resources at the community level. For example, planners will know the number of households without access to a vehicle such that alternative means of transportation can be arranged should a community require evacuation. Local planners can also use the database to determine the number of people in each census tract requiring an advisory in a language other than English. Other uses might include estimations of the amount of needed water supplies, food, medicine, and bedding; the number of emergency personnel required; the optimal locations for emergency shelters; the numbers of persons with special needs; and communities that will likely need ongoing support. The CDC SVI reports relative social vulnerability of census tracts and is therefore highly context dependent. As such, local expertise is critical to the application of CDC SVI in identifying vulnerable communities. As an example, in the case of Hurricane Sandy, we would expect local emergency officials to use their knowledge of the location of rental housing and of property values regarding areas bearing the brunt of the storm and resultant flooding to properly account for these potentially vulnerable households.

The CDC SVI or the underlying database is used in disaster management and in understanding the social determinants of health by federal, state, and local agencies, as well as private-sector organizations. For example, the CDC SVI is used to do the following:

• Map fire outbreaks and vulnerability metrics to target aid during emergencies (Lue and Wilson 2017).

• Plan hazard mitigation (Horney et al. 2015; Horney et al. 2017).

• Assess physical inactivity and fitness (An and Xiang 2015; Gay et al. 2016).

• Understand teen pregnancy rates (Yee, Cunningham, and Ickovics 2019).

• Assess human behavior and attributes, such as hazard awareness, rural–urban differences, migrant and refugee populations (Ikeda and Ozanne 2016), health status, and evacuation shelter deficiency (Karaye, Thompson, and Horney 2019).

• Assess the threat posed by climate change (Berke et al. 2015).

• Explore the association between heat-related outcomes and social vulnerability (Spangler et al. 2019; Lehnert et al. 2020).

• Assess social vulnerability with regard to the COVID-19 outbreak (C-SPAN 2020a, 2020b; Karaye and Horney 2020; National Institute of Environmental Health Sciences 2020; Nayak et al. 2020).

Concern 1: Data Representation

Specific questions and reservations pertaining to the article are as follows: Are FEMA’s Individual Assistance (IA) Program data the optimal representation of social vulnerability? If an area has been hit hard enough to reach “predetermined damage thresholds [that] can be declared federal disaster areas, enabling the flow of disaster recovery resources through FEMA’s Individual Assistance (IA) Program,” does that mean the physical impact is inherently sufficient to skew the entire population into a highly vulnerable state?

The authors stated, “We set analysis scope to encompass the set of tracts that both intersected the Sandy floodplain and had nonzero FEMA outcomes.” Do all of those eligible for FEMA’s IA apply for assistance? Do people living in socially vulnerable communities have the resources to navigate the bureaucracy of the IA system? Several articles in the popular press portray the lack of access that socially vulnerable populations have to FEMA funding. Examples include, “‘People just give up’: Low-income hurricane victims slam federal relief programs” (Vinik 2018) and “Black residents in Long Island community struggled to get federal aid” (Lane 2019). Peer-reviewed literature also notes an unequal distribution of federal resources; for example, “Unequal recovery? Federal resource distribution after a Midwest flood disaster” (Muñoz and Tate 2016). In view of these concerns, it would be useful to look at areas in the FEMA-defined flood zone that did not file claims or did not receive funding. Perhaps supplementing nonclaim flooded areas with remotely sensed imagery to verify storm damage would be useful in identifying truly socially vulnerable populations.

Concern 2: Analytic Methods

We also question the authors’ analysis methods. As they stated, “Figure 4 suggests that using correlations might be a misleading approach to validation, as convergence is stronger among the indexes than with the Sandy outcomes.” A standard statistical approach to assess convergence is through correlation. The analysis here demonstrates correlation among the indexes. Could the problem lie with the outcome variable instead of the indexes? Do FEMA’s IA data really capture the (intangible) construct of “social vulnerability”?

On request, lead author Rufat provided us with a tract-level file (N = 1,205) containing data used for the Rufat et al. (2019) analysis. The data fields in the table are as follows:

• Tract GEOID.

• Maximum flood depth.

• Percentage of applicants, affected renters, and damaged nonseasonal housing units.

• FEMA-verified losses in dollars.

• Percentile rankings for each of the four CDC SVI themes.

• Sum of the fifteen individual variable percentile rankings for the CDC SVI from which we calculated percentile rankings in Microsoft Excel using the PERCENTILERANK.INC function.

We noted inconsistencies between the data we received and the data the authors discussed in the article. Their article states that “models were constructed for all New York and New Jersey census tracts in the affected counties (N = 3,947).” Yet, the study area delineated in figures 1 and 3 of the article shows a greater geographic extent than the tracts in the data file provided by the author. Specifically, the figures show the full extent of sixteen counties in New York and New Jersey rather than the subset we received. Tracts in the provided file are within the surge extent defined by FEMA and the U.S. Geological Survey (although not all of the FEMA flood zone is in the study area). Because the authors did not provide all of the data used in re-creation of the CDC SVI model, we cannot compare figure 3 to any maps we produce from other models; for example, the Social Vulnerability Index. This discrepancy leads us to question the actual analysis scope: Is it the 3,947-tract area or the subset 1,205-tract area?

In addition, figure 1 shows a high-water depth of 6.8 m, but the maximum flood depth of tracts in the provided data file is almost 19 ft (about 5.8 m). Further, table 3 shows a maximum flood depth of 5.8 (unit unspecified). Why is there a difference in flood depth between figure 1 and table 3? Some of the tracts in the data file we received show a maximum flood depth of zero. Based on the original FEMA data for New Jersey, it appears that two tracts assigned zeroes in the provided table were instead flooded. Likewise, 201 New York tracts within the surge extent were assigned a flood depth of zero, even though the 3-m FEMA flood grid indicates flood depths greater than zero.

For the 1,205-tract database, the percentile ranks the authors calculated are largely in accord with our own calculations. Small differences might be due to rounding, but more germane is the authors’ lack of accounting for “no data” values. In the CDC SVI, where the census does not provide a value for a tract for any variable, we do not calculate a percentile ranking for that individual variable, or for its theme or overall ranking. Nine tracts in the 1,205-tract database should have been marked thusly, yet the authors assigned SVI scores for those tracts.

We differ with the authors on the notion that living in a densely developed area with high reliance on public transportation, such as Manhattan, should not be understood as a social vulnerability. We know from the 11 September 2001 tragedy just how socially vulnerable high-population, high-rise communities can be, especially when they have limited access to vehicles or routes of egress.

It is intuitive: Higher valued properties are generally found along ocean fronts, and poorer communities inhabit housing further inland. Thus, it is no surprise that the CDC SVI registered a negative association with housing damage inasmuch as Hurricane Sandy initially struck the coast of New York and New Jersey.

Concern 3: The CDC SVI Validation Objectives

Validity is the degree to which a test measures what it claims to be measuring. Validation of a social construct, here, social vulnerability, is complex because there exists no gold standard measure of social vulnerability. Not all social vulnerability indexes are constructed of the same cloth. Temporal and spatial focus will likely differ across the several indexes as well as by hazard type. These are important considerations as to the reason indexes might differ in results and measures of validity. Nevertheless, validation of the CDC SVI for a variety of use cases is ongoing, using both internal and external research (New York City Department of Health and Mental Hygiene and Office of Emergency Preparedness and Response 2013; Tarling 2017; Argonne National Laboratory and FEMA 2019). We use a framework described in “A Psychometric Toolbox for Testing Validity and Reliability” (DeVon et al. 2007), although we make no claim that this work is the definitive approach to establishing validity.

Commonly, in psychometric assessment, construct validity—the extent to which a tool, such as the CDC SVI, measures a construct it is designed to measure—is determined through several validation subtype methods. Translational validity, one validation subtype, indicates how well the measurement tool characterizes the construct. To achieve this type of validity requires the use of subject matter experts and literature reviews to make sound assumptions about the nature of the construct, in this case social vulnerability. The CDC SVI is based on subject matter expertise and extensive literature review. We contend that the CDC SVI has achieved translational validity.

Criterion validity, another subtype of construct validity, evaluates the correlation between the tool, in this case the CDC SVI, and a criterion, or outcome variable, representative of the construct. Criterion validity includes checks for concurrent, predictive, convergent, and discriminant validity. Concurrent validity indicates how the tool scores correlate with a related outcome at the same point in time. The tool scores might differ over time, but the related outcome should also differ relative to the tool scores. For example, does a higher CDC SVI 2016 score (i.e., greater vulnerability) mean a higher proportion of economic burden for a disaster in a comparable time frame, and does this relationship also apply for CDC SVI 2018? Predictive validity, as was undertaken by Rufat et al. (2019), denotes how well the tool forecasts a phenomenon it is expected to forecast. For instance, how well does the CDC SVI predict the ability of a community to recover after a disaster, or how many FEMA IA claims are filed?

Convergent validity reveals how different measures of the same construct correlate with one another. Does the CDC SVI correlate with the Social Vulnerability Index (Cutter and Morath 2013)? As an illustration of convergent validity, if a researcher creates a vulnerability index using a poverty variable and another creates a resilience index using a per capita income variable, we would expect much correlation between the two indexes on this variable. By contrast, discriminant validity is demonstrated when measures of theoretically unrelated constructs are, in fact, uncorrelated to one another. If the second researcher used area square miles as a variable instead of income, we would expect no significant correlation. To find a significant correlation in this instance indicates that we have not demonstrated discriminant validity. Such a finding would likely be cause for further investigation as to our variable choices. Within CDC, we have examined discriminant validity and found the CDC SVI has no association with a variety of theoretically unrelated variables. We have read studies indicating the CDC SVI correlates with other social vulnerability measures in some geographies (Lehnert et al. 2020) but not in others (Rufat et al. 2019). We have not yet examined concurrent validity.

As stated, we maintain that no single, tangible outcome representing social vulnerability exists. Therefore, criterion validity is difficult to achieve and can only be approached through the examination of outcomes for various use cases over place and time. The CDC SVI is best validated over the course of different hazard events (e.g., hurricanes, wildfires, epidemics) where the index was applied in conjunction with local public officials using ancillary data appropriate to the community and event. Importantly, although the CDC SVI was developed for ongoing application and is designed to be relevant over the entire hazard cycle, if it is to be used as a tool for reducing social vulnerability it is most useful as an advisory to knowledgeable local experts in the period preceding a potential hazard rather than in the immediate aftermath.

We recognize that the CDC SVI might not identify the most vulnerable populations in all applications and has not done so in the Rufat et al. (2019) study, with respect to outcomes. In this aspect, we agree with the authors. We suspect, however, that the reader might mistakenly conclude the validity of the CDC SVI is “poor” in all contexts, although this analysis is based on a single set of outcomes data for a specific event.

Through ongoing internal and external validation efforts, we will better ascertain the extent of practical applications of the index (New York City Department of Health and Mental Hygiene and Office of Emergency Preparedness and Response 2013; Tarling 2017; Argonne National Laboratory and FEMA 2019). We aim to publish results of these findings to inform on the scope and use of the CDC SVI in research and practice settings. The CDC SVI has evolved over time and will continue to do so. We have added auxiliary data providing daytime populations (because census variables are recorded at the “nighttime” domicile) as well as household health insurance status. Going forward, we expect to include housing tenure, which, along with health insurance, are two indicators that have gained prominence in recent years as markers of social vulnerability. Finally, because the CDC SVI and the accompanying database are freely available to the public, the tool is accessible by institutions, academics, and individuals interested in examining the social vulnerability of U.S. communities.

We thank the authors for providing this opportunity to reassess the CDC SVI, the concept and application of empirical validation, and the state of social vulnerability indexes in general.

Biography

BARRY FLANAGAN is a social geographer at CDC/ATSDR’s Geospatial Research, Analysis, and Services Program (GRASP), Atlanta, GA 30341. E-mail: bflanagan@cdc.gov. His research interests include the examination of structural determinants of health in the United States.

ELAINE HALLISEY is a geographer and geospatial analyst at CDC/ATSDR’s GRASP, Atlanta, GA 30341. E-mail: IMN7@cdc.gov. Her research interests include cartographic visualization, spatial problem solving, population health, and the interrelationships of geographic phenomena, both physical or environmental and human.

J. DANIELLE SHARPE is a geospatial epidemiologist at CDC/ATSDR’s GRASP and the Coordinator of the CDC Social Vulnerability Index (CDC SVI), Atlanta, GA 30341. E-mail: OYV7@cdc.gov. She is also a doctoral student in the Department of Epidemiology at Emory University. Her research interests include spatial epidemiology, social vulnerability, and geographic access to health care services.

CAITLIN E. MERTZLUFFT is a spatial epidemiologist with CDC/ATSDR’s GRASP, Atlanta, GA 30341. E-mail: IWE5@cdc.gov. Her research interests include examining the social and ecological determinants of health and health inequalities.

MARISSA GROSSMAN is a spatial statistician with CDC/ATSDR’s GRASP, Atlanta, GA 30341. E-mail: XMC9@cdc.gov. Her research interests include the environmental and social determinants of infectious disease.