The Four Domain Food Insecurity Scale (4D-FIS): development and evaluation of a complementary food insecurity measure

With the complementary measure the Four Domain Food Insecurity Scale, more women were classified as food insecure compared to the USDA Food Security Survey Module.

Abstract

The U.S. Department of Agriculture (USDA) Food Security Survey Module (FSSM) is a valuable tool for measuring food insecurity, but it has limitations for capturing experiences of less severe food insecurity. To develop and test the Four Domain Food Insecurity Scale (4D-FIS), a complementary measure designed to assess all four domains of the food access dimension of food insecurity (quantitative, qualitative, psychological, and social).Low-income Black, Latina, and White women (n = 109) completed semi-structured (qualitative) and structured (quantitative) interviews. Interviewers separately administered two food insecurity scales, including the 4D-FIS and the USDA FSSM adult scale. A scoring protocol was developed to determine food insecurity status with the 4D-FIS. Analyses included a confirmatory factor analysis to examine the hypothesized structure of the 4D-FIS and an initial evaluation of reliability and validity. A four-factor model fit the data reasonably well as judged with fit indices. Results showed relatively high factor loadings and inter-factor correlations indicated that factors were distinct. Cronbach’s alpha (ɑ) for the overall scale was 0.90 (subscale ɑ ranged from 0.69 to 0.91) and provided support for the scale’s internal consistency reliability. There was fair overall agreement between the 4D-FIS and USDA FSSM adult scale, but agreement varied by category. Findings provide preliminary support for the 4D-FIS as a complementary measure of food insecurity, with implications for researchers, practitioners, and policymakers working in U.S. communities.

Implications.

Practice: The Four Domain Food Insecurity Scale (4D-FIS) can be used to identify people who may be missed by more conservative measures like the FSSM and connect clients with available resources to mitigate food insecurity.

Policy: The 4D-FIS can be used to illuminate the range of experiences around food insecurity so policymakers can develop or modify local policies to address food insecurity.

Research: The 4D-FIS can be used to collect comprehensive data on food insecurity within local contexts and better understand how diverse experiences relate to diet and health.

Introduction

In 2019, 1 out of 10 households in the USA (10.5%) was classified as food insecure, meaning they did not have “consistent, dependable access to enough food for active, healthy living” [1]. National surveys suggest that rates of food insecurity have more than doubled during the COVID-19 pandemic [2, 3]. Research also shows that food insecurity is an indicator of lower subjective well-being and poor health [4–6], which underscores food insecurity as an important health determinant [7].

Since 1995, national estimates of food insecurity have been based on the U.S. Department of Agriculture (USDA) Food Security Survey Module (FSSM) [8, 9], a measure of the food access dimension of food insecurity [9]. The FSSM includes a full household scale (18 items) [10], a short household scale (six items) [11], and another scale (10 items) for adults or households without children [12]. The original intent of the FSSM was to measure and monitor changes in the prevalence and severity of food insecurity among U.S. households over time [8, 9]. Over the 25 years that the FSSM has been used for surveillance and research [8], the prevalence of food insecurity has remained relatively stable, fluctuating with changes in the economy but consistently remaining between 10% and 15% since 2001 [1]. (Although the USDA has not released a full report since the pandemic began, other studies suggest that the prevalence is now far higher [2, 3].) However, some research suggests that the FSSM may not be fully capturing shifts in food insecurity. For example, between 2009 and 2013, the prevalence of food insecurity as measured by the FSSM remained stable [1]. However, data from Feeding America showed an increase in the number of clients served by food pantries (from 37 million to 46.5 million) and in the prevalence of food insecurity among clients (from 76% to 86%) over the same period [13–15]. Maynard et al. argue that the FSSM is useful in providing standardized data but “may not accurately classify households and cannot provide insights into severity of food insecurity (thus potentially missing the opportunity to shed insights into those who are most vulnerable) (p. 12)” [16]. Especially now, given indications that food insecurity rates have skyrocketed during the COVID-19 pandemic [2, 3], it is critical to be able to accurately capture the range of experiences of food-insecure families.

The FSSM intentionally takes a conservative approach to measuring food insecurity, meaning that it focuses on more severe cases of food insecurity, where people are cutting or skipping meals, or not eating for an entire day [17–21]. In doing so, the FSSM may be capturing households that are the most food insecure but perhaps missing other households [21]. Research has established that the food access dimension of food insecurity includes four domains: quantitative (insufficient quantity of food), qualitative (inadequate food quality), psychological (uncertainty and worry about food), and social (social unacceptability) [9, 22–25]. However, the FSSM does not evenly represent all four domains. Rather, it focuses primarily on quantitative indicators of food insecurity, such as skipping meals, with less attention to qualitative indicators and little to no attention to social and psychological indicators like the sociocultural unacceptability of foods and feelings of deprivation, alienation, or shame [9, 16, 22, 26], which are most relevant for mental health [5, 16]. In practice, this means that the FSSM should accurately capture people at high levels of severity, who are reducing their food intake, but may miss people at low levels of severity, who are eating foods they deem unacceptable because of food shortages or because they are feeling stress, anxiety, or shame around running out of food. In addition, the FSSM requires three affirmative responses (in the 18- or 10-item scales) to be classified as food insecure [10, 12]. Some critics argue this cutoff may be too high given that some items have an acute focus on particularly harsh experiences like not eating for a whole day or eating so little that a person lost weight [20, 21].

The development and widespread use of the FSSM is an achievement and the USA has some of the most comprehensive data on food insecurity in the world [8, 9]. However, it is worth considering its limitations and developing complementary measures. As Maynard et al. note, we need more comprehensive measures of food insecurity to complement this data, providing a fuller picture of how the severity of food insecurity shapes health and well-being among U.S. individuals and households, in order to better inform research, programs, and policies [16]. Previous research (including the USDA’s reports, based on the FSSM) clearly demonstrates that food insecurity is more prevalent in households with children and in female-headed households, Black and Latino/a/x-headed households, Native American households, households headed by people with disabilities, and households with recent immigrants an noncitizen immigrants [1, 27–32]. We need to have comprehensive measures of food insecurity to understand how the severity of food insecurity impacts health, particularly for vulnerable and marginalized communities, who are more at-risk of food insecurity.

Given the limitations of the FSSM, this project developed and tested a complementary tool, the Four Domain Food Insecurity Scale (4D-FIS), to assess the four domains of food insecurity (quantitative, qualitative, psychological, and social) in a more comprehensive way. Study objectives were to (a) test the hypothesized structure of the 4D-FIS using confirmatory factor analysis, and (b) determine agreement for categorizing food insecurity between the 4D-FIS food insecurity scale and the USDA FSSM adult scale [12], drawing on a sample of low-income Black, Latina, and White women living in rural and urban areas of North Carolina.

METHODS

Study sample and recruitment

We used cross-sectional data from year 3 of a prospective project on childhood obesity that applied qualitative and quantitative methods to understand diet-related health disparities among low-income families in rural and urban areas of North Carolina [33]. Hardison-Moody et al. described recruitment in detail [33]. Participants were recruited in partnership with community-based organizations. Project staff developed a purposive sampling strategy that was locally representative of the low-income population of Black, Latino/a/x, and White subgroups. At recruitment (year 1), eligible participants were primary female caregivers (mothers and grandmothers) of at least one child ≥2 and <9 years of age with household incomes ≤200% of the federal poverty line. Slightly over half (51%) were classified as food insecure based on the FSSM scale at baseline. This sample represented women who were at-risk of food insecurity based on household characteristics [1], though eligibility was not based on food insecurity status. All participants provided written informed consent at the beginning of the project and again for the year 3 data collection. For recruitment, project staff attempted to recontact participants who had completed year 1 activities (n = 124). They succeeded at and locating many participants (n = 115). Three participants withdrew (n = 3), resulting in 112 participants for year 3. There was very little missing data. Of the 112 participants, three women had incomplete data and were excluded. This article includes 109 participants who completed the nutrition and survey interviews. Participants received $25 for each interview. Compensation was scaled to encourage retention, such that participants received a larger incentive at the final interview. The Institutional Review Boards at North Carolina State University and University of North Carolina at Chapel Hill provided approvals.

Data collection

The research team pre-tested data collection protocols, interview guides, and surveys with non-participants with similar sociodemographic characteristics to participants. Each participant completed three in-person interviews in this order: a sociological interview (open-ended questions focused on making ends meet, parenting, and health), nutrition interview (open-ended questions focused on the food choices, perceptions of food assistance programs, and food insecurity, including the 4D-FIS food insecurity scale), and survey interview (close-ended questions about household and participant demographics, income, and the FSSM adult scale [12]). Participants completed the three interviews on different days; ideally, they completed them within a 2-week period. The sociology and nutrition interviews were both semi-structured interviews, administered using interview guides and audio recorded. Interviews were completed in English or Spanish as appropriate according to participant preference. The nutrition interview used an interview guide that included the 4D-FIS food insecurity scale. The research team developed the 4D-FIS script to facilitate participants’ reporting of food insecurity, with framing text to introduce the set of 4D-FIS items and clarification text (if needed) to help with participants’ understanding of 4D-FIS items. Post-interview, interviewers created thumbnail summaries comprised of fieldnotes, emergent analytic themes, and reflections. While interview summaries were not formal qualitative analysis documents, summaries highlighted the most salient aspects of the interview.

Development of the 4D-FIS

A research team developed the 4D-FIS based on recommended scale development procedures [34]. Procedures included determining what to measure, generating an item pool, having the initial item pool reviewed by experts, administering scale to development sample, and evaluating the items [34]. The team consisted of women who were diverse in terms of race and ethnicity, socioeconomic status, place of origin, household family structure, academic discipline, and professional experience. Items were developed from a review of prior research [18, 23–25, 35–37] and existing food insecurity measures [10, 12, 38] and assessed these four domains: (a) quantitative, (b) qualitative, (c) psychological, and (d social [9, 25, 35, 36]. The 4D-FIS items were based on Hamelin and colleague’s core characteristics of food insecurity: shortage of food (quantitative), unsuitability of food and diet (qualitative), preoccupation or uncertainty in access to enough food (psychological), and alienation or lack of control over their food situation (social) [25], which are consistent with the four domains described elsewhere and recommended for use in future measures [9, 22]. While the FSSM was developed to assess all four domains, most of the FSSM items (six of the 10) focus on the quantitative (e.g., cutting meals) domain, with a couple of items for the qualitative (e.g., balanced meals), and one item for the psychological and social domains [17, 18, 20, 26] (e.g., feelings of deprivation or social unacceptability [9]). The 4D-FIS aligns theoretically with the FSSM but includes items for the psychological and social domains that are not covered by the FSSM. As recommended [34] and done in prior research [24, 35], food insecurity experts reviewed the items, evaluated the content validity, and provided comments. After additional revisions, the 4D-FIS was pre-tested with volunteers who were similar to the study participants. We used their feedback to modify the 4D-FIS before administration.

The 4D-FIS measures food insecurity at the individual (adult) level and not the household level. Prior research has documented the importance of separating individual-level experiences from household-level experiences of food insecurity [23]. The individual level was chosen because other important correlates or outcomes, such as dietary intake and weight, are measured individually. The reference period is the previous 30 days, because food insecurity experiences can vary within a monthly period, and a short time frame was needed to capture recent, salient experiences. The 4D-FIS is designed to be interviewer-administered to accommodate different literacy levels and the interview guide included additional text to facilitate participants’ understanding of items. To overcome participants’ reluctance to verbally acknowledge food insecurity, interviewers offered participants the option to physically (non-verbally) communicate their responses by pointing to a response option. Lastly, the 4D-FIS was designed to be relatively short (16 items, shown in Fig. 1) to minimize respondent burden.

Fig 1.

The Four Domain Food Insecurity Scale (4D-FIS). The 4D-FIS covers the four domains of the food access dimension of food insecurity: quantitative (3 items), qualitative (6 items), psychological (3 items), and social (4 items). For the quantitative, qualitative, and psychological items, response options are often, sometimes, rarely, and never. Social items used response options of agree a lot, agree a little, disagree a lot, and disagree a little. All items included blinded responses of do not know and refused. This scale is intended to be interviewed-administered and the 4D-FIS interview script is available by request from the corresponding author.

Scoring protocol for the 4D-FIS

The qualitative data, which were collected in the nutrition interviews, were used as a foundation for the scoring protocol. Given that the 4D-FIS was designed to capture women’s nuanced experiences of food insecurity, we aimed for the 4D-FIS scale to categorize participants consistent with how they described themselves in the qualitative data. While the 4D-FIS defines three conceptual categories like the FSSM (high, low, and very low food security), the 4D-FIS uses a different approach to determine severity. The 4D-FIS defines a person with no affirmative response as “food secure” and a person reporting any affirmative response as “food insecure,” as recommended by others studying food insecurity [21, 24, 38]. With the FSSM adult scale, food secure is defined by reporting two or fewer affirmative responses and food insecure is defined by reporting three or more affirmative responses [12]. In addition, the FSSM determines severity based on the total number of affirmative responses (the greater the number affirmative responses, the greater the severity of food insecurity) [12]. Previous research has suggested a progression of responses as food insecurity becomes more severe [24]. Worrying about food (a psychological experience) is one of the earliest indicators. As food insecurity intensifies, people are likely to first reduce diet quality or variety (a qualitative response) and then reduce food intake (a quantitative response) [24]. This research thus suggests that an affirmative response to the quantitative subscale indicates more severe food insecurity than an affirmative response to the non-quantitative subscales (qualitative, psychological, and social).

A two-step process was used to determine food insecurity status based on the qualitative data. In the nutrition interviews, participants were asked to recall current and former experiences related to not having enough food. First, the lead author systematically read all nutrition interview summaries (n = 109) and recorded observations that indicated one of the four domains of food insecurity, such as eating only once during the day, eating foods that participants considered inadequate (like bologna sandwiches, hot dogs, or cold cereal), and feeling frustrated, stressed, and left out due to constraints on food choices. Summaries varied in length; most were about five single-spaced pages, with some summaries exceeding 10 pages. Frongillo et al. used a similar approach to determine food insecurity status based on qualitative data [39]. Data from the food insecurity scales (the 4D-FIS or FSSM) were not used to make the determination. Summaries were reread to ensure consistent categorization. Exemplar cases were identified to identify participants who best represented each category. Second, we created an indicator variable for the qualitative categorizations and indicator variables for the 4D-FIS subscale scores. The subscale score was the sum of 4D-FIS indicators within each domain. Within each category, we examined the pattern of subscale responses from the 4D-FIS including the exemplar cases. Cut-offs were determined by the subscale responses within each category data from participants with valid data (n = 100). Data from nine participants were not used to develop the scoring protocol, because interviewers documented that the 4D-FIS data were inconsistent (n = 3) or understated (n = 6) compared to the qualitative descriptions in the nutrition interviews. This scoring protocol was applied to categorize all participants (n = 109) as food secure, mildly food insecure, or severely food insecure. Figure 2 presents the 4D-FIS scoring protocol.

Fig 2.

Scoring protocol for Four Domain Food Insecurity Scale (4D-FIS). This protocol was based on qualitative data from the nutrition interview summaries and responses to the Four Domain Food Insecurity Scale (4D-FIS). Responses of “often” and “sometimes” were counted as affirmative responses. Affirmative responses to the quantitative subscale were counted as more severe than affirmative responses to the non-quantitative (qualitative, psychological, and social) subscales.

Measures

Food insecurity was measured individually and retrospectively over the previous 30 days with two scales: (a) the USDA FSSM adult scale [12], and (b) the 4D-FIS. For the FSSM, categorical variables were created per standard protocol [12]. The FSSM determines severity based on the total number of affirmative responses (greater the number affirmative responses mean greater severity) [12]. The raw score is the total number of affirmative responses, where responses of “yes,” “often,” “sometimes,” and “3 days or more” were considered affirmative [12]. The raw score was categorized as follows: 0 = High food security, 1–2 = Marginal food security, 3–5 = Low food security, and 6–10 = Very low food security [12]. High and marginal food security were combined to form a food secure category; low and very low food security were combined to form a food insecure category [12]. For the 4D-FIS, items were re-coded as affirmative responses like the FSSM, where responses of “often” and “sometimes” were considered affirmative. The total number of affirmative responses were used to categorize food insecurity status, based on the scoring protocol developed for this purpose.

Confirmatory factor analysis

Confirmatory factor analysis is part of recommended scale development procedures [34]. Prior to completing the confirmatory factor analysis, item distributions (Supplementary Table 1) and inter-item correlations were examined [34]. Two items, hypothesized to relate to the qualitative domain, did not perform well (“How often did you eat plenty of healthy foods because there was enough food?” and “How often did you eat a wide variety of foods in the same meal because there was enough food?”). They were removed from the scale and subsequent analyses. Remaining analyses refer to the 16 retained items (Fig. 1). The confirmatory factor analysis evaluated whether the hypothesized four-factor scale fit the actual data using Mplus® software version 7.4 (Muthén and Muthén, Los Angeles, CA). Hypothesized relationships between the four factors and 16 indicators in the 4D-FIS were modeled as shown in Supplementary Figure 1. Each factor represented one domain of food insecurity, and indicators were the 4D-FIS items. The specified model allowed for correlated factors. All indicators were coded as binary variables to represent affirmative responses. The factor estimation method was for binary variables and mean- and variance-adjusted weighted least squares [40]. These fit indicators were used to evaluate if the data fit the hypothesized model: χ ² goodness of fit statistic, root mean square error of approximation (RMSEA), and comparative fit index (CFI) [40]. Cutoffs were established as: p > 0.05 (or ratio of χ ² to degrees of freedom [df] of <3:1), RMSEA ≤ 0.06 with 90% confidence interval (CI) lower limit near zero and upper limit below 0.08, and CFI ≥ 0.95) [40]. Factor loadings (>0.7) and inter-factor correlations (<0.9) were evaluated using recommended cutoffs [40].

Additional analysis

We calculated descriptive statistics and examined indicators of reliability and validity with SAS® software (Version 9.4 of the SAS System for Windows Copyright © 2002–2012 SAS Institute Inc., Cary, NC). Internal consistency reliability is the extent to which responses are consistent across indicators. We used Cronbach’s alpha (ɑ) to evaluate internal consistency for the 4D-FIS scale and subscales [34]. An ɑ value closer to one means that the set of indicators are closely related, ɑ > 0.7 is recommended [34]. To examine construct validity [34], we evaluated agreement between the 4D-FIS and the FSSM. Agreement was evaluated for three-level and binary categorizations. Kappa (ĸ) and p-values were calculated for each category and interpreted with established benchmarks: ĸ < 0 poor agreement, 0–0.2 slight, 0.2–0.4 fair, 0.4–0.6 moderate, 0.6–0.8 substantial, and 0.8–1 almost perfect agreement [41]. A non-significant p-value (p > 0.05) indicates agreement is not more than chance alone.

RESULTS

Sample characteristics are shown in Table 1. The sample was primarily young and middle-aged women (90% participants ≤50 years, range: 23–72 years). Women were racially and ethnically diverse: 43% identified as Black, 23% as Latina, and 33% as White. Fifty-one percent (n = 56) of participants reported financial struggles in the past year (e.g., not being able to pay rent or utilities), and 65% (n = 71) expected to experience hardship in the next 2 months.

Table 1.

Sample Characteristics for Women Living in North Carolina (n = 109)

Characteristic	n (%)
Individual
Age
Under 30 years	33 (30%)
31–50 years	65 (60%)
51 years and older	11 (10%)
Race/ethnicitya
Black	47 (43%)
Latina	25 (23%)
White	36 (33%)
Food insecurity statusb
Food secure	66 (61%)
Food insecure	43 (39%)
Household
Composition [mean (SD)]
Number of adults	2.2 (1.0)
Number of children under 18 years	2.3 (1.0)
Employment status
Yes (at least one member working for wages)	92 (84%)
No (no adult members working for wages)	17 (16%)
Total income from wages (dollars/month)	$1492 ($1354)
Participant in the Supplemental Nutrition Assistance Program
No	35 (32%)
Yes	74 (68%)

This sample of participants was from of a 5-year project conducted in rural and urban areas of North Carolina. Data shown are from female caregivers who participated in year 3 (n = 109).

^aOne participant reported more than one race/ethnic category (1%).

^bAssessed with the USDA FSSM adult scale. Food secure included categories of high and marginal food security; food insecure included categories of low and very low food security [12].

The four-factor model, evaluated in the confirmatory factor analysis of the 4D-FIS, had good overall model fit for three indicators. Results showed that the χ ² test p > .05 (χ ² = 94.4, df = 98, p = .6) and χ ²:df < 3:1 (1:1); RMSEA ≤ 0.06 and 90% CI lower limit near zero and upper limit below 0.08 (RMSEA = 0.00, 90% CI: 0.00, 0.05); and CFI = 1.0 [40]. Standardized factor loadings ranged from 0.72 to 0.98 (Table 2). Each factor had statistically significant associations (p < .05) with hypothesized items, and relatively large magnitudes (>0.7), which indicated stronger associations between the factor and items [40]. Inter-factor correlations ranged from 0.38 to 0.83 (Supplementary Table 2). All inter-factor correlations were below the recommended cutoff of 0.9 [40], which indicated that factors were distinct. There were positive relationships between factors, meaning that participants who reported more experiences in one domain reported more experiences in another domain. Regarding internal consistency reliability, the Cronbach’s alpha (ɑ) for the overall scale was 0.90, which exceeded the recommended cutoff of 0.70 [34]. Cronbach’s ɑ was 0.69, 0.79, 0.91, and 0.76 for the quantitative, qualitative, psychological, and social subscales, respectively (Supplementary Table 1). Data from Cronbach’s ɑ indicated scale and subscale items were closely related.

Table 2.

Standardized Factor-Item Loadings and Standard Errors from the Confirmatory Factor Analysis of the Four Domain Food Insecurity Scale (4D-FIS)

Item	Description	Quantitative	Qualitative	Psychological	Social	R  2
	Quantitative
1	Small	0.82* (0.07)				0.68
2	Ache	0.98* (0.06)				0.96
3	Hungry	0.93* (0.07)				0.86
	Qualitative
4	Important		0.91* (0.05)			0.83
5	Prefer		0.85* (0.05)			0.73
6	Canned		0.73* (0.09)			0.53
7	Same		0.92* (0.05)			0.84
8	No meat		0.74* (0.08)			0.54
9	Unsafe		0.72* (0.13)			0.52
	Psychological
10	Current			0.97* (0.02)		0.93
11	Future			0.98* (0.02)		0.96
12	Anxious			0.98* (0.02)		0.95
	Social
13	Control				0.87* (0.11)	0.76
14	Fair				0.68* (0.11)	0.46
15	Shame				0.92*(0.05)	0.84
16	Different				0.96* (0.04)	0.92

Factor loadings and standard errors were from the confirmatory factor analysis. There were four factors representing the four domains of food insecurity: quantitative, qualitative, psychological, and social. Indicators represent the 16 items included in the 4D-FIS. A factor loading with a magnitude > 0.7 is considered high [40]. The variance is shown in column labeled R².

*p < .05.

Participants who were classified as mildly food-insecure based on the qualitative data (n = 46) reported an average of two affirmative responses across all 4D-FIS subscales [mean (SD) = 2.0 (2.3)], with about one affirmative response to the qualitative [mean (SD) = 0.72 (0.96)] and social [mean (SD) = 0.74 (1.1)] subscales and rarely any affirmative responses to the quantitative [mean (SD) = 0.15 (0.36)] or psychological [mean (SD) = 0.37 (0.88)] subscales. This was a different pattern compared to those who were categorized as severely food-insecure based on the qualitative data. Those participants (n = 43) reported more affirmative responses across all subscales [mean (SD) = 7.3 (4.2)]. On average, participants categorized as severely food-insecure based on the qualitative data reported one affirmative response to the quantitative subscale [mean (SD) = 0.91 (1.1)], at least two affirmative responses for the qualitative [mean (SD) = 2.5 (1.9)] and social [mean (SD) = 2.1 (1.4)] subscales, and close to two affirmative responses to the psychological subscale [mean (SD) = 1.8 (1.3)]. Notably, only participants categorized as severely food insecure gave affirmative responses to the psychological subscale. Supplementary Table 3 presents descriptive statistics for the 4D-FIS subscales. Supplementary Figure 2 presents agreement between food insecurity scales and categorization based on the qualitative data. Results showed moderate (κ = 0.54) and slight (κ = 0.29) agreement for the 4D-FIS and FSSM, respectively, when comparing each scale to the qualitative categorization.

The 4D-FIS classified 32% (n = 35) of participants as food secure, 39% (n = 42) as mildly food insecure, and 29% (n = 32) as severely food insecure. In contrast, the FSSM classified 61% (n = 66) as food secure, 25% (n = 27) as mildly food insecure, and 15% (n = 16) as severely food insecure. For a three-level categorization, there was fair overall agreement between the 4D-FIS and the FSSM (overall ĸ = 0.31, p < .0001), which means there was higher agreement than could be expected due to chance. However, agreement was not the same for each category. There was fair agreement for the food secure (ĸ = 0.35, p = .0001), slight agreement for the mildly food insecure (ĸ = 0.13, p = .09), and moderate agreement for the severely food insecure (ĸ = 0.47, p < .0001) categories. In other words, the 4D-FIS and the FSSM agreed the most when categorizing households experiencing severe food insecurity, and they agreed the least when categorizing households experiencing mild food insecurity, with food-secure households in the middle. The non-statistically significant ĸ for the mildly food insecure category indicates that the agreement was not more than could be expected due to chance. For the binary categorization, results were like the three-level categorization. There was fair agreement between scales. Using the 4D-FIS, 32% (n = 35) of participants were considered food secure, and 68% (n = 74) were considered food insecure. With the FSSM, 61% (n = 66) and 39% (n = 43) were classified as food secure and food insecure, respectively. The binary categorizations appeared quite different (e.g., 4D-FIS categorized 68% participants as food insecure and the FSSM categorized 61% participants as food secure), as discussed below. The overall agreement was fair (overall ĸ = 0.35, p = .0001), which means the agreement was not due to chance alone.

Discussion

This study developed a scale, the 4D-FIS, as a complementary measure for assessing individual-level food insecurity. This scale was tested with a sample of low-income women. Results from the confirmatory factor analysis showed good overall fit with the hypothesized four-factor model. The relatively high factor loadings and inter-factor correlations provided evidence that the 4D-FIS assesses quantitative, qualitative, psychological, and social experiences of food insecurity. The first major finding is related to the initial evidence for the internal consistency reliability and constructs validity of the 4D-FIS. Cronbach’s alpha, an indicator of internal consistency, met the recommended cut-off for the overall scale, which suggests that the 4D-FIS items have strong relationships with the latent variable (food insecurity). These results are indicative of the scale’s reliability. We developed the 4D-FIS scoring protocol to align with the qualitative data, derived from semi-structured interviews with participants about their experiences of food insecurity. There was better agreement between the 4D-FIS and the qualitative categorization than between the qualitative categorization and the USDA FSSM adult scale. Overall agreement was fair when comparing categorization between both scales for three-level and binary categorizations. There were important differences between the two scales. For the three-level categorization, the two scales varied most categorizing participants in the less severe food insecurity category. For the binary categorization, the 4D-FIS categorized 68% participants as food insecure, while the FSSM categorized 61% participants as food secure. This difference is related to the two scales’ approach to scoring. The 4D-FIS categorizes people with at least one affirmative response as food insecure, which are a lower threshold for food insecurity than the FSSM (e.g., ≥3 items) [10, 12]. This cutoff (e.g., ≥1 item) is more consistent with the USDA’s broad definition of food insecurity and with previous studies that recommend defining food insecurity based on one indicator of food insecurity [21, 24, 38]. Coleman-Jensen similarly described a 70% increase in the prevalence of household food insecurity when using a less conservative definition than the FSSM definition [21].

As researchers recognize the need to broaden conceptions of food insecurity (beyond quantitative deprivation), in order to understand the experiences of underserved and vulnerable communities [22], comprehensive tools like the 4D-FIS can help to illuminate a wider range of experiences of food insecurity that other tools might overlook. The 4D-FIS attempts to capture experiences of food insecurity that might previously have gone undocumented using measures like the FSSM. However, this less conservative approach also means that more participants would be categorized as food insecure.

Second, our study demonstrates the importance of examining the full range of food insecurity experiences, particularly psychological and social experiences. The 4D-FIS included items for the psychological (n = 3) and social (n = 4) experiences. When we compared the patterns of responses to the 4D-FIS with the qualitative data from the interviews, evidence suggested that affirmative responses to the qualitative and social subscales of the 4D-FIS aligned with less severe food insecurity (as defined by the qualitative data), while affirmative responses to the quantitative and psychological subscales aligned with more severe food insecurity. Prior research has outlined a progression of food insecurity, in which the psychological experiences (e.g., worry) are associated with less severe experiences and the quantitative experiences (e.g., reductions in food intake) are associated with more severe experiences [24]. While our results supported social experiences (assessed in the 4D-FIS) as part of less severe food insecurity, our results did not support psychological experiences as part of less severe food insecurity. Instead, psychological experiences reflected more severe food insecurity. Review papers have provided evidence that food insecurity is associated with stress and depressive symptoms, especially for women and people from high-income countries like the USA [5, 16]. More research is needed to understand how the psychological and social indicators of food insecurity, traditionally considered less severe, may threaten well-being or mental health, especially for vulnerable subgroups [5, 16].

Based on our findings, the 4D-FIS has potential as a reliable and valid tool and can be tested more broadly. Health professionals (researchers and practitioners) need practical and flexible tools that can work in low-resource settings and with diverse populations [42, 43]. Like the FSSM, the 4D-FIS does not require a lot of time and can be administered in a variety of settings (e.g., homes and clinics). However, the 4D-FIS has distinct features that make this tool valuable for assessing food insecurity within specific communities or settings. First, the 4D-FIS includes framing text and options for non-verbal response options to facilitate administration with different samples and settings. Second, by including non-quantitative (in particular, psychological and social) domains of food insecurity, the 4D-FIS captures a wider range of experiences of food insecurity. For example, critics note that the FSSM fails to address feelings of deprivation and social unacceptability, which are part of food insecurity experiences [9, 22, 24, 25] and that the FSSM may overlook individuals or households dealing with less severe food insecurity [21]. Prior research points to a need for more comprehensive measures of food insecurity [9, 16], particularly at lower levels of food insecurity [21]. By assessing people’s experiences of food insecurity more fully, researchers can better understand relationships between food insecurity, diet, health, and well-being [4, 5, 16, 44], and practitioners and policymakers can obtain evidence needed to inform programs and policies for specific subgroups and settings [45].

This study had limitations related to the scale development procedures and to its reliance on cross-sectional data from a relatively small sample of women. The 4D-FIS was administered at one time to a development sample and not administered to a validation sample. The sample size (n = 109) was lower than the recommended sample size (n ≥ 200) for confirmatory factor analysis [34, 40], although the sample size met the minimum for a 16-item scale (e.g., 5–10 observations per item [40]) and scale development projects have been completed with sample sizes <200 [46]. The smaller sample represented the intended population of women who are at risk of food insecurity (e.g., women with children and women of color, from low-income and rural households [1]). Representativeness of both participants and settings is important for research to have real-world relevance [43, 47]. While it was advantageous that the sample was representative of women at risk of food insecurity, the sample was not representative of all adult subgroups at risk for food insecurity, in particular because it excluded men, which limits generalizability. This study did not include men because the larger childhood obesity project recruited only female caregivers [33]. Women (and mothers specifically) have traditionally been the focus of childhood obesity research. Given that men and women experience food insecurity differently [48], additional research is required to explore gendered aspects of food insecurity. Participants were recruited from only one region in the Southeast USA and this region is not representative of all communities. Future research is required to administer the 4D-FIS with larger and more representative samples and evaluate its reliability and validity.

Unique strengths of the study included the rigor in application of recommended scale development procedures, including the utilization of qualitative and quantitative data [34], purposeful sample of women at-risk of food insecurity [1], and the research team. Although cognitive interviewing was not possible [34], when developing the 4D-FIS and interview script, a diverse research team considered participants’ concerns and issues of linguistic and cultural relevance [34]. Additionally, the 4D-FIS was designed to be interviewer-administered to facilitate administration with participants. Cohen et al. discussed recognizing “invisible but essential” actions undertaken by research teams as an important part of translating research into real-world settings [42]. The 4D-FIS development process makes the tool potentially more valuable for assessing food insecurity for people in a range of demographic subgroups or settings who might be reluctant to share food insecurity experiences.

Translational implications

This research provided evidence for the 4D-FIS as a complementary measure for understanding people’s experiences of food insecurity. The results have implications for research, practice, and policy. Researchers should continue to use the FSSM for consistency in measurement and reporting and in large-scale efforts like state or national assessments. The FSSM is particularly valuable in tracking changes in the prevalence of food insecurity over time. Researchers should also consider using the 4D-FIS to collect more data that are comprehensive on food insecurity experiences in local contexts with specific subgroups and settings, especially in settings when it is important to identify the full range of people experiencing food insecurity. In future projects, researchers should consider collecting data on the administration of the 4D-FIS tool, so that others can learn about implementation in real-world settings [42]. Future research can advance understandings of how food insecurity in general or specific experiences (e.g., quantitative, qualitative, psychological, and social) relate to diet and health, particularly for vulnerable population subgroups [4, 5, 16, 44]. Practitioners can use the 4D-FIS, along with the FSSM, to better understand their clients’ needs, identify people who may be missed by more conservative measures like the FSSM, and connect clients with available resources to mitigate food insecurity including nutrition and food assistance programs. The 4D-FIS generates more nuanced data about the qualitative, psychological, and social domains of food insecurity, which can support practitioners as they design, implement, and evaluate more meaningful programs that are specific to people’s experiences, context, and cultures. Practitioners across fields (e.g., nutrition, social work, counselors) can use the data to partner, as part of translational team science, in the coordination of existing services or provision of new services to support clients’ food security and promote health [45]. For policymakers, data from the 4D-FIS offers a more comprehensive picture of food insecurity to aid in the development or modification of local policies. At the national level, the FSSM provides data required for surveillance, including that which informs federal nutrition and food assistance programs [8]. At local levels, the 4D-FIS can be advantageous for providing data specific to people’s experiences in that setting. The COVID-19 pandemic is changing how we consider vulnerability to food insecurity and chronic disease [49]. Emerging research suggests that the COVID-19 pandemic has exacerbated food insecurity among vulnerable subgroups (e.g., households with young children, Black and Latino/a/x-headed households) and increased food insecurity among people who had been previously unaffected [2, 3]. If the FSSM does not capture the full range of experiences and only identifies severe cases of food insecurity, we are missing opportunities to characterize and address food insecurity for the most vulnerable populations. A comprehensive measure can provide insights into diverse experiences of food insecurity and generate the actionable evidence needed to center health equity in research, practice, and policy.

Compliance with Ethical Standards

Conflict of Interest: No conflicts of interest declared.

Authors’ Contributions: C.M.J. conceptualized the study, analyzed the data, and prepared the first draft of the manuscript. C.M.J., S.E., A.H.-M., and S.K.B. assisted with data collection. All authors contributed to the study design, analysis, and writing.

Human Rights: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent: Informed consent was obtained from all individual participants included in the study.

Welfare of Animals: This article does not contain any studies with animals performed by any of the authors.