Diagnostic Accuracy of Two Food Insecurity Screeners Recommended for Use in Health Care Settings

Abstract

Objectives. To test the diagnostic accuracy of the American Academy of Pediatrics (AAP) recommended food insecurity screener.

Methods. We conducted prospective diagnostic accuracy studies between July and November 2016 in Chicago, Illinois. We recruited convenience samples of adults from adult and pediatric emergency departments (12-month recall study: n = 188; 30-day recall study: n = 154). A self-administered survey included the 6-item Household Food Security Screen (gold standard), the validated 2-item Hunger Vital Sign (HVS; often, sometimes, never response categories), and the 2-item AAP tool (yes-or-no response categories).

Results. Food insecurity was prevalent (12-month recall group: 46%; 30-day group: 39%). Sensitivity of the AAP tool using 12-month and 30-day recall was, respectively, 76% (95% confidence interval [CI] = 65%, 85%) and 72% (95% CI = 57%, 84%). The HVS sensitivity was significantly higher than the AAP tool (12-month: 94% [95% CI = 86%, 98%; P = .002]; 30-day: 92% [95% CI = 79%, 98%; P = .02]).

Conclusions. The AAP tool missed nearly a quarter of food-insecure adults screened in the hospital; the HVS screening tool was more sensitive.

Public health implications. Health care systems adopting food insecurity screening should optimize ease of administration and sensitivity of the screening tool.

Population health management requires health care providers to address the social and self-care needs of the populations they serve. Food insecurity, a condition of “limited or uncertain availability of nutritionally adequate and safe foods,”^1(p.1598) has been widely identified as a modifiable health-related social need.² To address food insecurity in the health care setting, health care providers need an efficient and valid strategy to identify and support individuals living in food-insecure households.

Food insecurity is a prevalent public health problem affecting 1 in 8 US households³ and is a highly stigmatized condition that is not commonly disclosed.⁴ Prevalence of food insecurity is highest among households with children: 1 in 5 of all households with children, 1 in 4 non-Hispanic Black households with children, and nearly 1 in 3 households with children headed by a single woman is food insecure.³ Food insecurity has been associated with costly health consequences for adults and children, including poorer physical and mental health and more frequent hospitalizations.^2,5–8 Documented consequences for children also include developmental problems and academic and social difficulty in school.^2,9,10

In 2014, the Institute of Medicine released recommendations for social domains that should be captured by electronic medical records, including food insecurity.¹¹ In 2015, the American Academy of Pediatrics (AAP) recommended that health care providers screen all households with children for food insecurity.¹² Studies in both ambulatory and hospital settings have shown that the majority of caregivers of children believe that it is appropriate for health care practitioners to address food insecurity.^4,13,14 Likewise, the majority of practitioners are willing to screen patients for food insecurity. Yet few practitioners actually do screen, citing a lack of valid tools and insufficient knowledge or resources to support families who screen positive.^4,13,15

The Household Food Security Survey (HFSS) is an 18-item, validated food security screening tool widely used in research (Table A, available as a supplement to the online version of this article at http://www.ajph.org).¹⁶ Although useful for research, the length and complex scoring algorithm limit routine use. The 6-item HFSS uses a subset of items from the 18-item HFSS and has high sensitivity and specificity (98% and 92%, respectively).¹⁷ Hager et al., in collaboration with Children’s HealthWatch, developed the Hunger Vital Sign (HVS), comprising 2 items from, and highly sensitive against, the 18-item HFSS (97% sensitive and 83% specific).¹⁸ The HVS was validated in urban emergency and primary care settings with predominantly non-Hispanic Black and Hispanic caregivers of children younger than 3 years. More recently, the HVS was tested for accuracy and performed well in a national population-based sample of adults that included multiple high-risk groups (≥ 97% sensitive; ≥ 74% specific).¹⁹

The AAP advocates an adaptation of the HVS, altering the items’ response categories to “yes” or “no” from the categorical response options used in the validated screen (“often true,” “sometimes true,” and “never true”).¹² Guidelines from the Centers for Medicare and Medicaid Services (CMS) also advocate a slightly adapted version of the HVS that keeps the same 3 HVS response categories but changes wording for consistency with other recommended screening items.²⁰ To our knowledge, the CMS adaptation has not been validated. Both the HVS and the AAP tools have been advocated for use by antihunger organizations and adopted by medical centers to identify food-insecure households and link them with resources (Appendix A, available as a supplement to the online version of this article at http://www.ajph.org). To our knowledge, a validation study of the AAP tool has not been published in the peer-reviewed literature.

The primary objective of this study was to test the diagnostic accuracy of the AAP tool against the 6-item HFSS (gold standard) for detecting food insecurity. The AAP tool uses a 12-month recall period. Working under the assumption that recent food insecurity may be more clinically relevant and actionable than an episode that may have occurred any time in the previous 12 months, we also tested the accuracy of a 30-day recall version of the AAP tool. A secondary objective was to test the diagnostic accuracy of the HVS (12-month and 30-day recall versions) in the same population. We hypothesized that simplification of the response categories in the AAP tool would result in lower diagnostic accuracy than the HVS and the HFSS for both recall periods.

METHODS

This study was conducted at an academic medical center in Chicago, Illinois. The primary service area (12 zip codes, 76 square miles) is home to 633 000 people: 77% non-Hispanic Black, 30% living below federal poverty level, and 22% unemployed.²¹ Previous studies estimate that the rate of food insecurity in the region exceeds national rates and is as high as 42% in families with children.²²

Participants

We recruited 2 convenience samples from the adult and pediatric emergency department waiting rooms between July and November 2016. Research assistants collected data during regular business hours. Eligible participants included those aged at least 18 years, patients, and other adults (i.e., parents and other caregivers, family members, and friends of patients). We excluded people exhibiting signs of distress or pain. Patients with high-acuity conditions bypassed the waiting room and were excluded.

Following informed consent, participants were given a survey and, upon completion, a small incentive (hand sanitizer or nail file). Participants were also given the contact information for a “community health information specialist,” a community health worker employed by the research team who could help locate food or other resources in their community.²³

Measures

Participants completed a 10-minute self-administered, paper-and-pencil survey eliciting basic sociodemographic characteristics, including age, education, race/ethnicity, marital status, and number of children younger than 18 years in the household.

12-month recall study.

Every questionnaire included 3 screening tools, all using a 12-month recall period: (1) the gold-standard 6-item HFSS, (2) the 2-item HVS (3 response categories), and (3) the 2-item AAP tool (yes-or-no response categories). The 6-item HFSS demonstrates 98% sensitivity in comparison with an 18-item version¹⁷ and was selected as the gold standard to minimize respondent burden. An affirmative response to 2 or more questions on the 6-item HFSS is considered a positive screen for food insecurity (Table A, available as a supplement to the online version of this article at http://www.ajph.org).²⁴ The HVS uses 2 items from the 18-item HFSS; 1 of these items is included in the 6-item HFSS and 1 is not. The shared survey item from the 6-item HFSS and the HVS was only included on the survey once. The 7 items comprising the 6-item HFSS plus HVS were administered in a single section of the survey. Using established criteria, we considered an affirmative response to either item on the HVS a positive screen.¹⁸ As recommended by the AAP, we considered an affirmative response to either of the 2 questions on the AAP tool a positive screen for food insecurity.¹²

30-day recall study.

The 30-day recall study used the screening tools as described in the previous paragraph, but with a 30-day recall period. In 1995, when the 18-item HFSS (12-month recall) was developed and tested, researchers also evaluated, but found less sensitive, a 17-item version with a 30-day recall period.²⁵ For the 6-item HFSS, we found no empirical assessment of the validity for a 30-day recall period, but the most recent version of the user guide suggests that it can be modified for a 30-day recall period.²⁴ We also found no peer-reviewed study of the validity of a 30-day recall version of the HVS. Our study validated the 30-day versions of the HVS and AAP tools by modifying the survey item phrasing from “Within the past 12 months” to “Within the past 30 days.”

Statistical Analyses

We used descriptive statistics to summarize sociodemographic characteristics and the prevalence of food insecurity for each study sample. We calculated sensitivity (percentage of those with food insecurity identified), specificity (percentage of those without food insecurity identified), positive likelihood ratios (probability of a food-insecure person screening positive divided by the probability of a food-secure person screening positive), negative likelihood ratios (probability of a food-insecure person screening negative divided by the probability of a food-secure person screening negative), and corresponding 95% confidence intervals (CIs) for the AAP and HVS tools with the 6-item HFSS as the gold standard. We tested for significant differences in the sensitivity of the AAP and the HVS tools by means of the McNemar test for paired categorical data. Because the risk of harm resulting from a false-positive screen for food insecurity is presumably lower than harm resulting from a false-negative, we identified sensitivity as the most important measure of diagnostic accuracy. We stratified sensitivity and specificity calculations by households with and without children.

Because priming participants with the HFSS and HVS before they responded to the AAP tool could bias diagnostic accuracy toward higher sensitivity, we randomized surveys to administer the same items ordered 1 of 2 ways: (1) AAP tool, sociodemographic characteristic items, then the HFSS and HVS items or (2) the HFSS and HVS, sociodemographic characteristic items, then the AAP tool. We calculated sensitivity and specificity by randomization strata for each sample.

Study results are reported following Standards for Reporting Diagnostic Accuracy Studies.²⁶

RESULTS

In the 12-month recall study, 188 of 255 eligible adults (74%) agreed to participate (Figure A, available as a supplement to the online version of this article at http://www.ajph.org). The majority of participants who declined did not provide a reason (57%); reported reasons included lack of interest (18%) and not feeling well (15%). Of the 188 participants, 154 had sufficient data to be included in this analysis; 32 participants were missing data for 1 or more of the food insecurity screening items, and 2 participants left the waiting room with their surveys. The majority of those with missing data responded “don’t know” to 1 or more of the survey items; only 6 people skipped 1 or more of the items. Participants (aged 18–73 years) were predominantly non-Hispanic Black (77%), single (54%), with some college or technical school (42%), and living with children younger than 18 years (51%; Table 1).

TABLE 1—

Characteristics of 12-Month Versus 30-Day Survey Participants: South Side Chicago, IL, 2016

Characteristic	12-Month Survey Participants (n = 154), No. (%)	30-Day Survey Participants (n = 118), No. (%)
Age range, y
18–34	56 (38)	56 (53)
35–64	83 (57)	42 (40)
≥ 65	7 (5)	8 (8)
Race/ethnicity
Non-Hispanic Black	102 (77)	83 (76)
Non-Hispanic White	10 (8)	12 (11)
Hispanic	9 (7)	6 (6)
Other or unknown	12 (9)	8 (7)
Marital status
Married	36 (23)	32 (29)
Single	83 (54)	67 (60)
Divorced or separated	24 (16)	10 (9)
Widowed	11 (7)	2 (2)
Education
Elementary or some high school	13 (8)	11 (9)
High school or GED	41 (27)	24 (20)
Some college or technical school	65 (42)	50 (42)
College graduate	35 (23)	33 (28)
Living with children aged < 18 y
Yes	76 (51)	66 (61)
No	74 (49)	43 (39)
Openness to information about food assistance when leaving hospital
Very open	. . .	32 (28)
Somewhat open	. . .	32 (28)
Not very open	. . .	18 (16)
Not at all open	. . .	33 (29)

Note. GED = general equivalency diploma. Because of some missingness, not all categories of data add up to total number of participants.

In the 12-month recall study, the prevalence of food insecurity was 46% according to the 6-item HFSS (the gold standard), 39% with the AAP tool, and 53% with the HVS (Figure B, available as a supplement to the online version of this article at http://www.ajph.org). The sensitivity of the AAP tool was significantly lower than the HVS (76% [95% CI = 65%, 85%] vs 94% [95% CI = 86%, 98%]; P = .002; Figure 1). The specificity of the AAP tool was higher than the HVS (93% [95% CI = 85%, 97%] vs 82% [95% CI = 72%, 90%]). The positive likelihood ratio of the AAP tool was similar to that of the HVS (11 [95% CI = 5, 23] vs 5 [95% CI = 3, 8]). The negative likelihood ratios were also similar: 0.3 (95% CI = 0.2, 0.4) for the AAP tool compared with 0.1 (95% CI = 0.0, 0.2) for the HVS. Of those who screened negative with the AAP tool but positive with the HVS tool, 92% screened positive because they selected “sometimes true” for 1 or both HVS survey items.

FIGURE 1—

Sensitivity and Specificity of the Hunger Vital Sign and the American Academy of Pediatrics Recommended Tool: South Side Chicago, IL, 2016

Note. AAP = American Academy of Pediatrics; CI = confidence interval.

The sensitivity of the AAP tool was slightly higher among households with children (78% [95% CI = 61%, 90%]) compared with households without children (71% [95% CI = 52%, 86%]), but the sensitivity of the HVS was the same for these groups (94% [95% CI = 81%, 99%] vs 94% [95% CI = 79%, 99%]). The sensitivity of the AAP tool was lower when administered after the 6-item HFSS and the HVS (71%; 95% CI = 54%, 85%) and higher when it was administered before (82%; 95% CI = 65%, 93%).

In the 30-day recall study, 154 of 218 eligible adults (71%) agreed to participate (Figure A). Among those who declined, the majority (56%) did not report a reason; reported reasons for refusal included lack of interest (20%) and not feeling well (16%). Of the 154 participants, 118 had sufficient data to be included in this analysis; 34 participants were missing data for 1 or more of the food-insecurity screening items, and 2 participants left the waiting room with their surveys. Participants (aged 18–86 years) were predominantly non-Hispanic Black (76%), single (60%), with some college or technical school (42%), and living with children younger than 18 years (61%). When asked if they would be open to receive information about food assistance when leaving the hospital, 82% of those with food insecurity and 39% of those without were somewhat or very open (Table 1).

In the 30-day recall study, the prevalence of food insecurity was 39% according to the gold standard 6-item HFSS, 31% with the AAP tool, and 46% with the HVS (Figure B). The sensitivity of the 30-day recall AAP tool was significantly lower than the HVS (72% [95% CI = 56%, 84%] vs 91% [95% CI = 79%, 98%]; P = .02; Figure 1). The specificity of the 30-day recall AAP screen was 96% (95% CI = 88%, 99%), and the specificity of the HVS was 83% (95% CI = 73%, 91%). The positive likelihood ratio of the AAP tool was similar to that of the HVS: 17 (95% CI = 6, 53) versus 5 (95% CI = 3, 9). The negative likelihood ratios were also similar for the AAP and HVS tools: 0.3 (95% CI = 0.2, 0.5) versus 0.1 (95% CI = 0.0, 0.3). Of those who were missed by the AAP tool but captured by the HVS, 90% screened positive because they selected “sometimes true” for 1 or both HVS survey items.

In contrast to the 12-month recall study, the sensitivity of the 30-day recall AAP tool was lower among households with children (67%; 95% CI = 46%, 83%) compared with households without children (79%; 95% CI = 49%, 95%). Similarly, sensitivity of the HVS was lower among households with (93%; 95% CI = 76%, 99%) compared with households without children (100%; 95% CI = 78%, 100%). As seen in the 12-month recall study, the sensitivity of the AAP tool was lower when administered after the HFSS and the HVS (67%; 95% CI = 46%, 83%) and higher when it was administered before (79%; 95% CI = 54%, 94%).

DISCUSSION

Because food insecurity is recognized as a prevalent and modifiable condition that can impede the public’s health and medical care,⁷ health care systems are beginning to adopt screening and food support interventions into their clinical workflow. This study evaluates the diagnostic accuracy of the AAP recommended food-insecurity screening tool and the effect of the recall period on the tool’s diagnostic accuracy. We found that, in an urban population with a high prevalence of food insecurity (46% vs 13% nationally),³ the 12-month and 30-day recall versions of the tool recommended by AAP lacked sensitivity, failing to detect more than a quarter of individuals with food insecurity. Sensitivity of the tool did not vary significantly by recall period. These findings have important implications for adoption of the AAP-recommended yes-or-no response tool.

This study also evaluated the diagnostic accuracy of the widely used HVS tool for both the 12-month and 30-day recall periods. We found that, in contrast to the AAP tool, both the 12-month and 30-day recall versions of the 2-item HVS were highly sensitive in this population, detecting at least 93% of food-insecure households with and without children. The sensitivity of the HVS tool using 12-month recall in our population was similar to that recently reported for a large, nationally representative sample.¹⁹ Although we found the HVS to have higher sensitivity than the AAP tool, the specificity was lower (82% vs 93%). The positive and negative likelihood ratios—the likelihood that a positive screen indicates food insecurity and a negative screen indicates food security—were similar, although our study likely lacked power to detect small differences. For both tools, the 12-month recall version was more sensitive than and at least as specific as the 30-day recall version. Although the CIs overlapped, the 12-month HVS had the highest sensitivity overall for food insecurity in this population. Given the well-documented negative consequences^2,5–10 and availability of interventions to address food insecurity,^27–30 a screening tool should optimize sensitivity (reduction of false negatives) over specificity (reduction of false positives).

Routine screening for food insecurity is feasible in the health setting^4,31,32 and can enable support for food and other health-related basic needs.^22,29,30 Wilson and Jungner’s widely accepted criteria for effective screening require that the test be acceptable to the population.³³ In this study, cooperation rates with self-administered food insecurity screening were high. There were no participant complaints about any of the screening tools and item nonresponse rates were very low. The AAP’s recommendation to simplify the validated HVS to yes-or-no response options was likely intended, at least in part, to increase provider acceptability and adoption. Other self- and provider-administered screening questions typically invite a yes-or-no response (e.g., Are you sexually active? Do you wear a seatbelt? Have you had a flu shot?). Although easier to administer, yes-or-no questions, especially for stigmatized conditions, may be more prone to measurement error than questions with polychotomous response options.³⁴ Our findings suggest that social desirability bias may explain variation in diagnostic accuracy between the HVS and the AAP tool: endorsing “yes” to food insecurity may be more stigmatizing than endorsing the “sometimes true” category used by the HVS. For both recall periods, more than 90% of those captured by the HVS but missed by the AAP tool screened positive because they selected “sometimes true” for 1 or both survey items. Patient and provider acceptability and mode of administration will be important considerations for optimizing both the accuracy of food insecurity screening and fidelity to a screening protocol.

Wilson and Jungner’s criteria also stipulate that screening should only be implemented with a policy and intervention in place to treat the condition. The AAP, in partnership with the Food Research and Action Center, offers an online toolkit for providers to address food insecurity, including screening (the toolkit recommends the HVS), connecting patients to food and nutrition services, and supporting policies related to reducing food insecurity (http://frac.org/aaptoolkit).³⁵ In the 30-day recall study, we queried participants’ openness to receive information about food support from the hospital. Although people with food insecurity were twice as likely to be open to this information, so were nearly 40% of those who were food secure. In a recent study of caregivers of children presenting for well-child care, when given the opportunity via a self-complete questionnaire, 54% of food-insecure caregivers requested referrals for food support (e.g., finding food pantries). Moreover, 21% of food-secure caregivers requested food support referrals.³⁶ As suggested by our previous work, these food-secure individuals may value information about resources to help family members or friends²³ or may be food secure because they are utilizing similar resources and need additional resources to maintain food security.

Limitations

Our findings should be interpreted in the context of several limitations. This study used a convenience sample of adults at 1 health care organization site. The sample may differ from other patient populations, and therefore the diagnostic accuracy of the AAP tool may differ, positively or negatively, in those populations. It may be especially important to further examine the diagnostic accuracy in caregivers of pediatric patients. We found no difference in sensitivity between households with and without children for either the AAP or HVS tool; however, CIs were wide. Caregivers of children may be especially reluctant to self-identify as food insecure because of fear of negative consequences (e.g., report to child protective services); for this population, the response option of “sometimes” may be especially crucial.

Food insecurity screening in this study was self-administered and more than half of the population had at least some college education. It is possible the diagnostic accuracy of the AAP and HVS tools would vary if administered orally or in a population with less education. Although we did find a statistically significant difference in the sensitivity of the validated AAP versus the HVS tool, CIs were wide. Because health systems are moving quickly to adopt food insecurity screening, we opted for the shortest possible enrollment period to achieve our primary study aims. Even if we doubled the sample size, the estimated effect on the CIs for sensitivity and specificity estimates would be modest (±3% points for the 12-month recall study). Lastly, although not a limitation, we note that we did not see a significant priming effect according to the order in which the AAP tool was administered. The sensitivity of the AAP tool was actually slightly, but not significantly, higher when administered before the gold-standard HFSS rather than after; the sensitivity was still lower than that of the HVS.

Public Health Implications

In this predominantly non-Hispanic Black, urban population with high rates of food insecurity, the HVS was highly sensitive and performed better than the AAP tool for identifying food-insecure households with and without children and for both recall periods. In our children’s hospital alone, which admitted approximately 3700 patients in 2016, adoption of the AAP tool would have missed identification of at least 250 and as many as 600 food-insecure households. For screening in unique populations and for other proposed adaptations of the HVS (such as the screener recently proposed by CMS),²⁰ the validation steps described in this study can be replicated before expending resources on provider training and adoption into the workflow. Organizations that adopt food-insecurity screening should also enact policies and practices to effectively intervene.