Challenges to evidence-based health promotion: a case study of a Food Security Coalition in Ontario, Canada

Summary

Developing the evidence base for health promotion can be challenging because interventions often have to target competing determinants of health, including social, structural, environmental and political determinants; all of which are difficult to measure and thus evaluate. Drawing on a case study of food insecurity, which refers to inadequate access to food due to financial constraints, we illustrate the challenges faced by community-based organizations in collecting data to form an evidence base for the development and evaluation of collective programmes aimed at addressing food insecurity. Interviews were conducted with members of a multi-stakeholder coalition (n = 22 interviewees; n = 10 organizations) who collectively work to address food insecurity in their community through a range of community-based programmes and services. Member organizations also provided a list of measures currently used to inform programme and service development and evaluation. Data were collected in a city in Southern Ontario, Canada between May and September 2015. Participants identified four barriers to collecting data: Organizational needs and philosophies; concerns surrounding clientele wellbeing and dignity; issues of feasibility; and restrictive requirements imposed by funding bodies. Participants also discussed their previous successes in collecting meaningful data for identifying impact. Our results point to the challenge of generating data suitable for developing and evaluating programmes aimed at broader determinants of health, while maintaining the primary goal of meeting clients’ needs. Documenting change at intermediate- and macro-levels would provide evidence for the collective effectiveness of current programmes and services offered. However, appropriate resources need to be invested to allow for scientific evaluation.

BACKGROUND

There is a growing recognition of the importance of evidence-based practice in health promotion. It has been shown to lead to more effective interventions, and provides organizations with the information necessary to evaluate impact (Green, 2000, 2002; Rychetnik and Wise, 2004). However, producing the evidence to inform evidence-based health promotion can present challenges given that interventions include activities to address determinants of health (social, structural, environmental and political) that are difficult to measure (Green and Kreuter, 2005) and are beyond the day-to-day mandate of many community organizations. Community interventions often target a variety of drivers (social, environmental, political), and thus there are a multitude of considerations that need to be accounted for in their development and evaluation (Green, 2006).

Ideally the provision of health promotion interventions requires acknowledgment of how social contexts shape individual and community circumstances (Kickbusch, 1989). Interventions thus require coordination across a range of organizations and various levels and targets for intervention. It is this coordination that leads to challenges in the development and evaluation of health promotion interventions. Consequently, health promotion continues to struggle with its evidence base (Van den Broucke, 2012) due to what Koelen et al. (2001) refer to as dilemmas in: Interventions and outcomes (difficulty identifying independent and dependent variables related to social, cultural and organizational factors within complex organizations); numbers (difficulty in measuring outcomes at social, political and environmental levels); and the absence of control groups. Tilford (2000) argues that while the importance of disseminating evidence in health promotion is well-recognized, ‘appropriate methods to enhance uptake and use of evidence have not always been applied’ (p. 661). The current paper presents a case study identifying challenges faced by community-based organizations in collecting and disseminating data to determine if their existing programmes and services are working to address the underlying determinants of food insecurity.

THE CASE

The case explored in this study focuses on responses to household food insecurity within one community. Food insecurity exists ‘when people do not have physical and economic access to sufficient, safe and nutritious food to meet their dietary needs and food preferences for an active and healthy life’ (Dietitians of Canada, 2005). In 2012, the last year for which a nationally representative estimate is available, almost 13% of Canadian households were characterized as food insecure, representing over 4 million affected Canadians. The correlates of household food insecurity have been well-documented through analyses of national survey data and include reliance on government assistance as the main source of income, being in a household led by a lone mother, Aboriginal status, and lack of home ownership (Che and Chen, 2001; Ledrou and Gervais, 2005; Kirkpatrick and Tarasuk, 2009, 2011; Tarasuk et al., 2012). Food insecurity has been recognized as a serious public health problem given the links with compromised nutrition and poor physical, mental and social health (Vozoris and Tarasuk, 2003; Health Canada, 2007; Kirkpatrick and Tarasuk, 2008; Mikkonen and Raphael, 2010; Dietitians of Canada, 2016; World Health Organization, 2016). Responses to food insecurity have been dominated by food banks and other food relief programmes, as well as capacity building initiatives such as community kitchens (Dietitians of Canada, 2016). However, there have been repeated calls for responses that address the underlying determinants. For example, a 2016 position statement from Dietitians of Canada calls for responses to address the root causes of the problem (Dietitians of Canada, 2016). Within this context, local communities are attempting to craft strategies that go beyond providing food relief as a means of ameliorating vulnerability to food insecurity. However, generating evidence of the effectiveness of interventions targeting underlying determinants (e.g. housing, employment) is challenging, and requires appropriate targets and methods for data collection and analysis.

Our research was conducted in a Southern Ontario, Canada community where 12 local organizations have self-organized to develop a multi-sectorial Food Security Coalition (FSC) to address this growing issue within their community. Collectively, they offer a wide range of health promotion services that aim to address the determinants of food insecurity by focusing their services at drivers of food insecurity, such as employment and lack of housing. Local food banks also participate in the FSC, although their role is more an immediate response to food insecurity rather than addressing the broader determinants. Through engagement with our participating community as part of a larger research initiative, it was identified that the community faces significant challenges in collecting data to form an evidence base for both developing and evaluating the impact of the FSC in ameliorating vulnerability to food insecurity. Whilst data exist in other jurisdictions, this community (along with others) has unique characteristics that make it difficult to determine if what ‘works’ in other jurisdictions will have a similar (if not negative) impact in the context of this community. Working alongside the FSC, we designed a research project to interrogate these challenges and develop a means by which the community might work to collect meaningful evidence within the given constraints. The project aim was to map out key actors and the infrastructure necessary for building a more comprehensive evidence base for developing and evaluating interventions targeting food insecurity in this community.

METHODS

Our research was designed using a case study methodology. Case studies are research strategies in which the distinguishing characteristic is an attempt to examine a contemporary phenomenon in a real-life context (Yin, 1981). The methodology enabled us to uncover challenges faced when collecting the data required to develop evidence-based interventions and demonstrate programme effectiveness. We adopted a two-step approach that incorporated developing an inventory and semi-structured interviews (Crabtree and Miller, 1999). The inventory allowed us to identify the types of data organizations are already collecting. The interviews allowed us to understand, from the participating organization’s perspective, the difficulties faced in collecting data. Collectively, our data provide an understanding of the additional actors, and infrastructure necessary for building a more comprehensive evidence base for developing and evaluating interventions targeting food insecurity.

Recruitment

Individual organizations were recruited through the Chair of the FSC, with whom the authors met through a larger research initiative in the case city. Representatives from the FSC organizations were contacted by the Chair through e-mail, sent a letter of consent and an interview time was scheduled based on the participants’ schedule and preference. The research team interviewed 22 individuals (representing 10 organizations). Two FSC member organizations did not participate; a charitable organization and an organization offering mental health services. Interviews were conducted in person or by telephone by J.B., S.E. and S.M. between May and September 2015.

Data collection

Data inventory

Participating organizations were asked to provide the team with surveys or other instruments they use to collect information from their clients, or to list the type(s) of data they track regarding service use. An inventory spreadsheet was created to organize this information. The aim was to understand if and how organizations on the ground were collecting data that permitted them to measure their impact on community food insecurity.

Interviews

Semi-structured interviews sought to uncover the range of approaches taken, and challenges faced, by FSC member organizations when collecting data to develop an evidence base. Participants were asked to discuss the factors that influence what data they collect from their clients, and any related barriers or challenges they may face when collecting or disseminating this data to member organizations. We wanted to understand what the organizations identified as necessary to developing a comprehensive evidence base for developing and evaluating interventions targeting food insecurity in their community.

Interviews were transcribed verbatim. Steps used for analysis followed those outlined by Miles and Huberman (1994): (i) coding of the transcribed data to identify the words or sections of text regarding challenges faced in collecting data from clients; (ii) sorting through the coding to identify themes with regard to challenges faced and (iii) confronting these themes with a formalized body of knowledge (researcher’s expertise and academic/grey literature). Data analysis was carried out by three independent researchers (J.B., S.E. and S.M.). Discrepancies were settled during two full days of discussion regarding emerging themes and structure of the current paper. Data analysis was also informed by ongoing consultation with the FSC with whom the researchers have been engaging with since June 2014. Data were discussed with the FSC on two separate occasions (April and November 2015) through oral presentations, whereby themes emerging from the data were confirmed as being representative of the voices of committee members. All interviewees reviewed the data presented for accuracy.

For the purpose of anonymity, we have assigned a label to each participant based on their organizational mandate. If there were multiple organizations that represented a category, the organizations were further denoted with a number. Several of the participating organizations were represented by numerous workers/volunteers, and each of these participants is denoted with a letter.

Ethics

This research was approved by the University of Waterloo Office of Research Ethics (project number 20748). All participants provided written consent.

RESULTS

Data inventory

The data currently collected by members of the FSC provide member organizations with information about the socio-demographics of their clientele, frequency of service use, physical and mental health, finances and use of social services (Table 1). While this information may be used to tailor programmes to suit specific client needs, and to observe changes in these variables over time, it does not necessarily provide a sufficient evidence base for developing interventions or evaluating existing services’ impact on food insecurity. The limitations of these data, as prescribed and restricted by a number of different factors, were discussed during our interviews.

Table 1:

Examples of data collected by FSC member organizations

Address	Record of self-harming behaviour
Name/identification	Referral source provided
City	Relationship with parents
Community of origin/postal code	Resources referred to
Dietary considerations	School
Emergency contact	Statement of desire programme outcome
Employment and/or schooling	Age
Extent of drug/alcohol use	Anonymous visits
Family doctor	Date of birth
First and last name	Debt levels
History of police involvement	Expenses (monthly)
Housing status/household makeup	Grade
Income source	ID presented
Involvement in counselling	Income
Living arrangements	Number of people in family/children
Marital status	Number of visits
Names of children or dependents/birth dates	Permission to contact parent
Parents name	Pregnant/nursing
Personal contact	Return visit (or new visitor)
Phone number	Sex
Presence of physical/mental health concerns	Thoughts of self-harm
Reason for visit

Interviews

The main themes identified in the interviews relate to: organizational needs and philosophies; concerns surrounding clientele wellbeing and dignity; issues of feasibility; and restrictive requirements imposed by funding bodies. Interviews also highlight some of the successes and potential opportunities related to enhancing programme and service delivery that may also inform future debates around: What data is most meaningful to collect; what data might best inform an evidence base; and what data might accurately reflect the impact of interventions.

Organizational needs and philosophies

While the organizations’ emphasis is direct service rather than evaluation, they do seek information from clients for the purpose of improving service delivery. Participants noted that demographic and/or anecdotal information allows them to better understand the needs of different population groups and adapt services to meet those needs. For example, in recent years, Food Bank 2 has seen an influx of single men using their services, with families being more likely to frequent Food Bank 1. Two of the food banks emphasized that having the flexibility and autonomy to make judgment calls over the kind of data, if any, they collect is imperative to ensuring they are sensitive to the unique circumstances of their clients and can subsequently tailor services in an individualized fashion. As noted by Family/Financial Assistance 1-C, certain segments of the food insecure population ‘only go to the one food bank because they [the organisation] don’t ask questions and that’s one way of [the clients] keeping control of their own information’. Food Bank 3-A discussed that organizations took diverse approaches to serving the community, reflecting that although other organizations require more invasive information, their mandate is to first ask clients ‘what can we do to help?’ rather than asking for their personal information. This is of particular interest, given that numerous organizations discussed the fact that access to healthy food is a fundamental human right, and should be treated as such.

Informal conversations and interactions with clients were valued as an effective approach for informing food bank staff about the needs of clients, in comparison to more rigid processes of data collection and intake. Two volunteers at Food Bank 2, a faith-based organization, summarized this position:

Our role is that if somebody comes to the door and says they are hungry, we provide food because that is what we’re called to do by our faith. It is as simple as that (Food Bank 2-A). It is as simple as that. (Food Bank 2-C)

Because the drivers of food insecurity are complex and many factors influence the financial resources available to a household for food and other basic needs, many FSC organizations collect data consistent with their specific mandate; data that may be used to inform interventions aimed at the broader determinants of food insecurity (e.g. financial counselling), among other social issues. For example, family services requests information regarding income versus monthly living expenses to tailor programmes to their clients, while the women’s shelter tracks information-like client engagement with counselling services (Women’s Shelter-A; Women’s Shelter-B). However, currently there is no infrastructure for identifying what impact these services are having on rates of food insecurity.

For other organizations, an intentional choice was made not to collect data as part of their service provision. For example, eligibility and access to the school food programme is not tied to, or dependent on, demonstrated indicators of poverty or food insecurity. The school board places importance on minimizing the stigmatization of students who are ‘in need’. Collecting data conflict with philosophical values of preserving client confidentiality and dignity. This concern was echoed by other participants, for example, Food Bank 1 who identified the intrusive nature of collecting and tracking client information:

[if we were going to collect more data from clients] I’d have to know more of what and who was going to be involved and who is going to see this information. I mean it’s just like I said, if I was John or Fred, or whoever I was, I don’t think I would like to have people knowing everything about me.

Concerns surrounding clientele wellbeing and dignity

Participants expressed widespread concern over requiring clients to provide information that could give the impression that their service use was being monitored, as this could be both a stigmatizing action and a perceived infringement on privacy rights. NGO-C noted that they observe a common hesitance across the food banks to collect data that might suggest ‘they gave [food] to someone who didn’t deserve it’. There was apprehension that the data could be used, or even be perceived as being used, to identify clients who are violating rules on how frequent one is allowed to access a food bank within a prescribed timeframe. This sentiment was asserted by food bank representatives. For example, Food Bank 3 explained that they register people without ‘going through their circumstances’, and simply ask for some personal information and photo ID, although this is not a prerequisite to access the food bank services. Ultimately, even if clients say “‘well, I don’t have any ID’, nobody ever goes away [hungry]…we never say no” (Food Bank 3-A).

Several participants alluded to the importance of maintaining the dignity of the clients they serve, and their concern that collecting data from clients in the form of intake tools and registration forms may detract from this objective. For example, Food Bank 1 said that they would not want to ask for more data from clients because they think that it ‘would be very intrusive’ (Food Bank 1-A):

They have probably gotten to a point where some of them may be awfully tired of having to keep telling their stories. You don’t ever want to give the impression that we are judging or evaluating them, because they are very sensitive to that. (Food Bank 2-A)

The fear of further victimizing already marginalized or isolated segments of the population was a large concern voiced by participants.

…most of the people who come to our food bank already feel that Big Brother is watching them all the time – they’re tired of having to justify their reasons for having to attend a food bank. That’s sort of what we try to stay away from. We want to build them up, not make them feel less secure than when they walked through the door. Food Bank 2-B

Food Bank 2 said that food banks should be a place where people can come freely without pressure or the obligation to disclose personal information—a ‘no questions asked’ place where they can simply access food (Food Bank 2-C). This theme emerged throughout the interviews with food provision service providers and is well summarized by Food Centre-A:

We (try and) track new participants in the meals, so we can have an idea of discrete users rather than just a number. That’s the only time when we’re really asking people to just sort of sign in and it’s really only to track, it’s not to qualify for what we do here. We would never be screening to make sure the people qualify, make sure they are ‘poor enough’. It’s voluntary – every time someone does give us information, other than when we track them as a number, it’s voluntary. We communicated to them that they’re helping us do the work that we’re doing here by participating in our surveys.

This reality is part of the reason that Food Bank 3 ‘switched [from having a conventional administrator] to having greeters that would take that moment to talk to [clients]’ in order to signal a welcoming environment. Adopting a more conversational tone in collecting information and greeting individuals represents a concerted effort to de-stigmatize the emergency food provision service. However, Food Bank 3-A stressed the reality that this ‘modest attempt’ in probing deeper into people’s lives is often unsuccessful due to the fact that clients are often hesitant to discuss their personal circumstances in public. Nevertheless, many did indicate that they saw value in having more information relating to demographics, such as more detailed employment data, to help illuminate root causes of food insecurity and other poverty related problems. While this information might not specifically inform the food bank services because they are already aware that the individuals they see are food insecure, this information might benefit other member organizations whose focus is to address the broader determinants of food insecurity.

Issues of feasibility

Reliance on an under-staffed office and poor infrastructure were also voiced as a significant challenge to collecting data for the purpose of evaluation. For example, Family/Financial Assistance Services 1-A voiced concern over the inability to evaluate the more intangible benefits of their services, such as broader community or economic benefits, due to feasibility concerns and their limited human capital:

We were talking about the evaluation of social value…[but] we have a very slim infrastructure. We have a beautiful building, but we don’t have an IT person, we don’t have an HR person, we don’t have a janitor!

Another key concern noted that relates to feasibility was the inherent information-gathering restrictions that come with a reliance on volunteerism to operate and deliver services. The inability to adequately train and retain volunteers is certainly a well-documented challenge in the not-for-profit sector. Food Bank 2-A expressed reservations that even if they wanted to collect more information, they are not in a position where they would have the capacity to do so. Food Bank 2-B said that ‘if it was a paid position, I would probably approach it with a more analytical [lens]…that’s a lot of work when you add it up…it’s staggering – we’re talking thousands of hours’. Organizations are clearly conscious of not over-burdening their volunteers with data collection. As Food Bank 2-A explains, ‘if we did it in a more elaborate and sophisticated way, we wouldn’t be able to have volunteers do what they do – we wouldn’t have as many volunteers.’ The lack of administration hinders the organization’s ability to collect data. Additionally, front-line service providers may not be the best individuals to elicit sensitive personal information, as they have to be ‘really economical about what [they] ask and how [they] ask it’ (Food Bank 2-A). This sentiment was well-supported by NGO-C, who stated that it is not the FSC’s intention to ‘add an administrative burden to people who are already often overwhelmed with administrative tasks. We don’t want to take people from dealing with clients beyond their expertise levels, or to filling [out more] forms.’

Restrictive requirements imposed by funding bodies

The type of funding that is acquired by the participating organizations ranges from local funding from wealthy philanthropists (e.g. food banks), to national not-for-profit funders (e.g. school food programme), to Federal or Provincial funding (e.g. Family/Financial Assistance-2). The majority of the organizations track data to justify their role in the community and need for funds to various funding bodies, or to simply provide evidence of the number of clients they are currently and/or prospectively serving. The data are collected by the organizations to demonstrate both ‘need’ and adherence to programme targets and/or organizational mandates. Many participants, particularly those who come from organizations that report to Provincial and National funding bodies, argued that much of the data they feel compelled to collect does not inform or benefit their day-to-day operations, with the exception of potentially assisting in their prospects for securing future funding. For example, Food Centre Program-A characterizes the information that their organization collects as ‘pretty much a funding funnel’, with little ‘real purpose to asking any of it’ in practice. Women`s Shelter-B similarly proclaimed that the information they track ‘doesn’t really influence the day-to-day work [and] most of the time is… very time consuming’. Community Food Centres Canada-A’s funders require them to track the number of educational materials and publications they produce, and how they have been distributed, which is ‘not something that we would ever care to track.’ School Food Program-A expressed similar sentiments:

I am working on my site visit reports today that I did all year and I look at them now, and I looked at my notes, and I think ‘Why am I keeping this? Is this of any use to me now?’

Given that the data many organizations are required to collect for funders is perceived as irrelevant to their frontline service to clients, there was some frustration that this takes away from the time and capacity available for carrying out other actions that may directly benefit their clients, particularly in the short term. Several organizations noted that they funnel data up to Provincial or National funding or reporting bodies and expressed hope that the data could be used to lobby the government for more funding or to influence policy change (e.g. Family/Financial Assistance 1-1, Family/Financial Assistance 2-A and Women’s Shelter-B). In a climate of limited resources and capacity, local organizations felt they could benefit from greater clarity around whether or how the information they are encouraged to collect is actually being used at higher levels to promote meaningful policy change. Community Food Centres Canada-A states:

Some of the information would really tell them [funders] nothing about our [actual] impact. To achieve more substantive change when it comes to poverty, we need to be looking more at the impact of the supports and services that exist, and maybe speaking more honestly with people that are trying to live on those, and trying to understand how they impact their lives. It’s just something we don’t have the resources to do at the time, but we’re certainly trying to get funding to look more carefully at [how] our advocacy office and social justice program impact people that come through our doors. I think it’s a matter of looking more carefully at the root causes of the poverty and how we could better address those.

This narrative also alludes to the necessity to embrace more qualitative approaches to data collection in order to adequately illustrate the true impact of the services provided. In this vein, the local development NGO is seeking to strengthen what they call the ‘new frontier’ of both qualitative and quantitative data collection, especially pertaining to programme outcomes (NGO-B). The type of information they want to collect would supplement existing datasets, while providing a deeper understanding of outcome-related findings. The NGO cites the desire to expand ‘evidence-based action’ through the elevated collection of ‘much more primary data that proves (an appropriate) use of funds’ (NGO-B). However, there is a lack of adequate funding and resources required to implement such initiatives.

Successes in collecting meaningful data for informing interventions

The interview data also revealed means through which FSC members are working to collect data that may inform evaluation of the collective efforts of individual organizations targeting broader determinants of food insecurity. Information that was perceived as most valuable for evaluation purposes was identified as going far beyond that which is typically collected for the purpose of securing funding. For example, the health unit collects higher level data such as average rents, whether their clients are receiving social assistance, etc., in order to assess the pertinence of the food baskets they distribute and to assess who could afford to live in the local community with all these factors considered. This approach appears to be embedded within an evolving trend to better understand the overall context of food security by asking about drivers of food insecurity (e.g. cost of living). For example, the NGO whose primary function is mobilizing and funding other local organizations to address identified community needs is ‘moving into a different model of measuring outcomes for the people they support’; yet is in early stages of the process of instituting this change (NGO-1). One characterization of this evolution is the FC’s approach, which includes the unique incorporation of pre-and-post client data that asks qualitative questions such as: ‘has the FC [food centre] increased your access to healthy foods?’ This information is combined with personal anecdotes on how the programme impacts a client’s life quality to collectively comprise the bulk of their annual report (Food Centre-A). Similarly, Family/Financial Assistance 1 aims to collect information that helps to reveal a deeper picture of the individual’s life and lifestyle:

For example, ‘do you smoke?’, do you, ‘yes, but I am going to quit, I am trying to quit’ and then there’s always a story. So, you do get a lot of personal information in, that comes with one question. So, the stats that we, all that information gets plucked into the system - every single thing - now not [emphasis added] that all the personal or the, the stories that may come… Which are very helpful because that gives me the information I needed if it’s something that I can offer than I can refer them to the programs or the services they need to help them do that.

Changes within the aforementioned organizations is reflected in, and supported by, Canadian Food Centres Canada (CFCC) who has been heralded by participants as helping to signal the trend towards more nuanced ways of gathering and reporting information related to food security, food access, and life quality. Within the organizations they govern, the CFCC encourages the introduction of programme-specific data collection that contain less demographic information, and more information relating to attitudinal and behavioural changes around food (Community Food Centres Canada-A). The Food Centre is one of the CFCC projects, therefore shaping the way they operate and the information they collect. Certain Food Centre programmes require unique sets of information—some examples include: food harvested, meals served, food sent home and recipes tried (Food Centre-A). Food Centre-A vocalized what working with the CFCC has allowed them to do:

If you can have the numbers to show for what you do, that it’s going to just benefit you […] The first thing that they have done is set up an online tracking system, and every programmer has the responsibility to track levels of participation and also their outcomes. We [track] the number of volunteers who helped out…for our social justice and activism it’s basically who participated and an intake form that summarize every case that’s come through.

In this model, staff can encourage the inclusion of quotes and stories; however, outcomes are still primarily based on quantitative data (Food Centre-A).

DISCUSSION

Health promotion interventions are often required to target a variety of underlying drivers (social, environmental, political), making it difficult to identify independent and dependent variables for measurement (Koelen et al., 2001). Determining the effectiveness of such interventions is challenging, and there are a multitude of considerations that need to be accounted for in their development and evaluation (Green, 2006). Food insecurity is one such example that illustrates these challenges. As demonstrated in the interviews and by the development of the FSC, food insecurity is being embraced as a priority for a number of overlapping—yet incongruent—environmental, health promotion and social equity agendas, in recognition of the complex drivers that influence financial resources for food. Such an integrative approach to practice can foster collective action to address complex health issues, while taking into account the issues related to inequity and sustainability (Lundy, 2010). This approach gets at a much broader definition of food insecurity than the focus on insufficient resources at the household level. Through our case study however, we illustrate a number of difficulties faced by front-line providers in collecting the data necessary to develop evidence-based interventions and more centrally, demonstrate the effectiveness of their programmes in addressing the broader determinants of food insecurity in their community. As the data illustrate, there are challenges to building an evidence base for community-based organizations that require coordination across multiple organizations and settings, who have unique priorities, constraints, and agendas.

Of greatest concern to participating organizations was ensuring that any data collected did not stigmatize or further victimize clients. Although it is widely supported that food security can be enhanced by having effective data collection, monitoring, and evaluation mechanisms (McCullum et al., 2005), Brug et al. (2010) caution that strict metrics often adversely impact organizations who require more individualized interventions for their clients. This may relate specifically to the food banks and community food centres, where the aim is not to address complex drivers of food insecurity, but rather to provide a service for those in immediate need. However, participating organizations beyond those related to food provision have also advocated for a qualitative approach to data collection, whereby client’s experiences could be used more extensively to evaluate programmes—this based on the idea that the success of the interventions can be determined by whether the needs of the clients are met. However, this does not tell us if the broader determinants, rather than immediate needs, are being addressed. Evaluation methods such as contribution analysis may be particularly useful for understanding how clients attribute their experiences, or outcomes, to the intervention (Mayne, 2001). In this type of evaluation, a theory of change is established to outline how community-based programmes and services should lead to specific outputs or outcomes. Contextual data can then be collected to validate or modify this intended theory. Indeed, qualitative methods are important to this evaluation approach by contributing a rich picture of processes, achievements, and conditions for improvement (Koelen et al., 2001). Kemm’s (2005) discussion of ‘what constitutes evidence’ suggests that health promotors have long argued for a broader definition of ‘evidence’ that includes narrative descriptions and ‘lay knowledge’. Noting the limitations of narrative descriptions and views of practitioners and recipients (e.g. causation, validity), he argues that the question should drive the methods. It is evident from our data that questions around the impact of programmes on the lives of clientele may be better answered through narrative, though consideration needs to be given to how these data might be collected and the infrastructure required for tracking and analysis.

Our data also suggest that collective action creates challenges for implementing evidence-based practice and that individual level data cannot account for the variety of factors that shape socially determined health outcomes. Instead, our results point to the need for community-level longitudinal measures, including but not limited to robust quantitative measures of food insecurity, that can document the impact of FSC member organizations’ interventions. Documenting change at the macro-level, rather than exclusively at the level of specific clientele, would provide evidence for the collective effectiveness of current programmes and services being offered. However, appropriate resources would need to be invested in to allow for scientific evaluation, coupled with ethical clearance for data collection; a barrier not noted by participants. If the FSC were provided with the resources necessary for enhancing data collection and maintenance capacity (i.e. sustainable infrastructure and administrative support), their front-line efforts could remain focused on meeting the more immediate needs of their clients. This approach would alleviate the need for individual organizations to collect their own data, and perhaps if coordinated appropriately, higher-level data could document community-level change to demonstrate need to funding bodies. Furthermore, the development of community-level measures could incorporate input from both front-line providers and, when appropriate, clientele who may be the best source of input for identifying what data to collect, and through what practices. Although we present this as a ‘simple’ solution, we recognize complex health issues require time, realistic goals, and complex methods for measurement. Indeed, it is well established that interventions that require significant broader change require a far longer commitment than behavioural interventions and, additionally, are harder to demonstrate as ‘successful’ due to their inherently complex nature (Brownson et al., 1999; Golden and Earp, 2012). Consideration would also need to be given to the maintenance of client confidentiality and to ensure that programme evaluation does not hinder the intentions of the organizations. In presenting this approach we recognize that calls to expand the support for multi-level interventions have largely gone unheeded, increasing the challenges faced by health intervention practitioners (Golden and Earp, 2012). However, decisions must still be made and actions taken even when our science or evidence is deemed imperfect (Funtiwicz and Ravetz, 1993; Edge and Eyles, 2014).

CONCLUSION

It is critical to population health that we generate knowledge around the effectiveness of health promotion interventions, including documenting why they were, or were not, successful. Greater reliance on evidence has much to offer in terms of improving the effectiveness of interventions, the quality of health promotion practice, and achieving the most effective allocation of resources (Green, 2002). However, determining the effectiveness of particular health promotion interventions may require other types of evidence (Learmonth and Mackie, 2000; Dooris, 2005) and need to be informed by concepts of scientific evidence that are more suited to health promotion (Rychetnik and Wise, 2004; Aro et al., 2005; Armstrong et al., 2007). The perverse consequence of the inability of health promotion to provide evidence of effectiveness using the established methodology of evidence-based medicine may well have weakened its position within health policy, as policymakers tend to take the incorrect but often-held view that a lack of evidence of effectiveness equals evidence of no effect (Speller et al., 2005). The desire to measure the evident ‘success’ of interventions continues to dominate much of the discourse on health promotion effectiveness. Rather than pose solutions, we hope that this case study has illustrated the challenges faced by the front-line providers whose primary intention is to serve their interests of their clients and their community, and points readers in the direction of how to make effective change.