Implementing an Evidence-based Tobacco Control Program at Five 2-1-1 Call Centers: An Evaluation Using the Consolidated Framework for Implementation Research

Abstract

Introduction

The Smoke-Free Homes (SFH) Program is an evidence-based intervention offered within 2-1-1 information and referral call centers to promote smoke-free homes in low-income populations. We used the Consolidated Framework for Implementation Research to conduct a mixed-methods analysis of facilitators and barriers to scaling up SFH to five 2-1-1 sites in the United States.

Methods

Data were collected from staff in 2015–2016 via online surveys administered before (N = 120) and after SFH training (N = 101) and after SFH implementation (N = 79). Semi-structured telephone interviews were conducted in 2016 with 25 staff to examine attitudes towards SFH, ways local context affected implementation, and unintended benefits and consequences of implementing SFH.

Results

Post-implementation, 79% of respondents reported that SFH was consistent with their 2-1-1’s mission, 70% thought it led to more smoke-free homes in their population, 62% thought it was easy to adapt, and 56% thought participants were satisfied. Composite measures of perceived appropriateness of SFH for 2-1-1 callers and staff positivity toward SFH were significantly lower post-implementation than pre-implementation. In interviews, staff said SFH fit with their 2-1-1’s mission but expressed concerns about intervention sustainability, time and resources needed for delivery, and how SFH fit into their workflow.

Conclusions

Sites’ SFH implementation experiences were affected both by demands of intervention delivery and by SFH’s perceived effectiveness and fit with organizational mission. Future implementation of SFH and other tobacco control programs should address identified barriers by securing ongoing funding, providing dedicated staff time, and ensuring programs fit with staff workflow.

Implications

Smoke-free home policies reduce exposure to secondhand smoke. Partnering with social service agencies offers a promising way to scale up evidence-based smoke-free home interventions among low-income populations. We found that the SFH intervention was acceptable and feasible among multiple 2-1-1 delivery sites. There were also significant challenges to implementation, including site workflow, desire to adapt the intervention, time needed for intervention delivery, and financial sustainability. Addressing such challenges will aid future efforts to scale up evidence-based tobacco control interventions to social service agencies such as 2-1-1.

Introduction

Smoke-free home policies reduce exposure to secondhand smoke.^1,2 Although such policies were reported by 83% of US households, lower rates of smoke-free home policies are found in African-American and low-income households, as well as households with a smoker.^3–5 Such policies are important because secondhand smoke exposure is associated with a host of negative health outcomes.³ Approximately 21% of nonsmoking adults in the 2010–2011 National Health and Nutrition Examination Survey demonstrated recent secondhand smoke exposure based on blood levels of serum cotinine; exposure was much higher among children ages 3–11 (41%), non-Hispanic African-Americans (47%), and people living in poverty (43%).⁶

The Smoke-Free Homes (SFH): Some Things Are Better Outside brief intervention was designed to reduce exposure to secondhand smoke in the homes of these priority populations. SFH was developed and evaluated in a pilot study,⁷ followed by an efficacy study,⁸ and two replication studies.^9,10 These studies showed that SFH increases the prevalence of in-home smoking bans and reduces secondhand smoke exposure in low-income households in which one or more residents smoke.^7–10 In this project, five organizations in four states delivered SFH for at least 1 year. Delivering SFH involved five steps: (1) screening 2-1-1 callers for eligibility; (2) enrolling eligible callers, who completed an initial assessment; (3) mailing printed intervention materials at three time points; (4) delivering one telephone coaching call; and (5) administering a telephone survey (for evaluation purposes) after the intervention had been delivered. A web-based tool guided program delivery, including recruitment scripts, coaching messages, and telephone survey items. The research team provided training and technical assistance but was not involved in delivering the program.

SFH research—including this implementation evaluation—has been conducted in partnership with 2-1-1, a network of information and referral helplines in the United States and Canada connecting users with local health and social service resources. Most 2-1-1 users are poor and seek help paying for rent, utilities, and food.^11–13 They smoke at roughly twice the rate of the US population and have lower rates of smoke-free home policies.^14–16 Nationally, 2-1-1s receive about 17 million calls per year through over 200 local and statewide contact centers and have the potential to reach millions of low-income smokers.¹⁷

The current demonstration project was undertaken to analyze real-world implementation and potential scalability of SFH. If evidence-based interventions are to be scaled up from research settings to broader implementation, it is critical to understand factors that influence implementation and dissemination.^18,19 This mixed-methods study examined the challenges and benefits of integrating SFH into the daily work of five 2-1-1 sites. Implementation experiences were evaluated based on the Consolidated Framework for Implementation Research (CFIR),²⁰ which assesses characteristics of an intervention, the outer setting (ie, the broader context in which the organization is situated), the inner setting (ie, aspects of the organization that affect implementation), characteristics of individuals involved in implementation, and the implementation process (ie, planning, engaging, executing, and evaluating). This study was designed to inform future efforts to scale up tobacco control programs by addressing three research questions:

1. What were staff attitudes towards and beliefs about the SFH program, and how did they change over time?

2. Did sites adapt the intervention, and to what extent did the local context of each 2-1-1 affect implementation?

3. What unintended benefits and consequences did 2-1-1s experience as a result of integrating the SFH program into their work?

Methods

In 2014, a competitive grant application process was used to select sites. Emails describing the project were sent to 2-1-1s that had worked with the research team previously. Information was also provided via a listserv to US 2-1-1 staff. Staff at 31 sites submitted letters of intent, which were rated by the research team based on potential impact, magnitude of local need, institutional support, and program delivery plan. Eleven finalists submitted full applications including a budget, timeline, and detailed plans for delivering the program and enrolling participants. The research team reviewed applications and selected five sites (see Table 1) with the strongest priority impact scores. Program implementation occurred June 2015–October 2016. The team leading this implementation evaluation did not play a leading role in developing SFH or conducting any of the prior effectiveness studies.

Table 1.

Description of the five 2-1-1 call centers participating in the implementation study of the Smoke-Free Homes (SFH) Program

Site	Call volume	FT/PT staff	Site description	Description of callers	Participants enrolled in SFH
A	78000	6/2	Single call center in an urban county in the East North Central region of the Midwest, operating 2-1-1 and other social service programs	Primarily urban, half African-American, half white, mostly women and with incomes below 200% of federal poverty level	546
B	131000	15/14	Statewide collaboration of nine call centers with regional coordinators in each of four area codes in an East South Central state. Individual call centers implemented SFH.	Primarily African-American and women, about two-thirds urban and one-third rural	605
C	279000	17/12	Single call center in the East North Central region of the Midwest, serving 15 counties with a population over 2 million.	Predominantly women, both urban and rural, half African-American, 41% white	490
D	150000	6/40	Largest single call center in one South Atlantic state (and among the 10 largest call centers in U.S.)	Predominantly female, 39% African- American, 32% white, 28% Hispanic	335
E	143000	11/8	Single call center serving 37 counties in a West South Central state	Predominantly female, 54% white, 28% African-American, 8% Native American	369

Call volume is annual, rounded to the nearest 1000. FT/PT = full-time/part-time. Level of staff participation ranged from asking a single screening/eligibility question to being centrally involved in all phases of the project. Data come from sites’ applications, which were submitted in 2014 in response to a request for proposals.

Data Collection

All staff involved in SFH tasks (mailing, recruiting, coaching, follow-up interviewing, and managing) were asked and given multiple reminders to complete three surveys between June 2015 and November 2016: one pre-training (N = 120), one post-training (N = 101), and one post-implementation (N = 79). Surveys were completed online using Qualtrics (Provo, UT, 2015). Once post-implementation surveys were complete at each site, a subset of staff selected based on SFH roles participated in in-depth qualitative interviews (September–November 2016). Surveys and interviews were completed before intervention effectiveness results had been shared with sites.

Staff did not receive incentives for completing surveys or interviews. Washington University’s institutional review board approved this study.

Measures

Survey items were created by the study team based on CFIR domains and constructs,²⁰ as well as concepts from Proctor et al.’s implementation framework, including feasibility, acceptability, and sustainability.²¹Table 2 provides information about surveys and the interview guide.

Table 2.

Sample survey questions and response options for the pre-training survey, the post-training survey, the post-implementation survey, and the structured qualitative interview guide, all based on constructs from the Consolidated Framework for Implementation Research (CFIR)

Data collection tool	Sample items	Underlying CFIR constructs	Response choices
Pre-training survey (19 items)	I am clear about the steps of the SFH Program. In my personal opinion, it is very important for our 2-1-1 to offer the SFH Program.	Characteristics of individuals: Knowledge and beliefs about intervention Inner setting: Compatibility	Strongly disagree (SD) to strongly agree (SA), don’t know (DK) SD to SA; DK
Post-training survey (28 items)	Being able to adapt the recruitment and delivery of the SFH Program is very important to our 2-1-1. Overall, my personal learning objectives have been achieved.	Intervention characteristics: Adaptability Characteristics of individuals: Other personal attributes	SD to SA; DK SD to SA; DK
Post-implementation survey(34 items)	The communication among our SFH Program team members was excellent. How easy or difficult was it to adapt the program to your 2-1-1? How comfortable or uncomfortable were you delivering the SFH Program coaching intervention?	Inner setting: Networks and communications Intervention characteristics: Adaptability Characteristics of individuals: Individual stage of change	SD to SA; DK Very difficult to very easy; DK Very uncomfortable to very comfortable; DK/not applicable
Interview guide (number of items variable based on staff role)	Did your 2-1-1 encounter any unintended negative consequences from participating in the SFH Program? What advice would you give 2-1-1s considering such partnerships?	Process: Reflecting and evaluating Process: Reflecting and evaluating	Open-ended Open-ended

The pre-training survey assessed perceptions of 2-1-1 site culture and understanding of and beliefs and attitudes about SFH.

The post-training survey assessed reactions to the training and post-training beliefs and attitudes toward the intervention.

The post-implementation survey assessed SFH implementation. Five items that appeared in both the pre- and post-training surveys were repeated (rephrased in past tense); these were used to form the positivity index (see below). Three items that appeared in the post-training survey only also were repeated and used to form the appropriateness index (see below).

Composite implementation measures

Two composite measures were created to assess change over time in implementation attitudes and perceptions. The first was a positivity index created by summing responses to five items assessing staff attitudes toward SFH that were included at all three time points (based on CFIR domains of intervention characteristics and inner setting). The index included the following items: The enrollment goals for SFH are realistic; I believe implementing SFH will be difficult (reverse scored); I believe that delivering SFH will take too much of my time (reverse scored); I do not believe we have the resources needed to be successful in delivering SFH (reverse scored); and I think this program will lead to more smoke-free homes in our population. Response options on a 5-point scale ranged from strongly agree to strongly disagree. Responses were recoded as ordinal (2 = strongly agree for positively worded items or strongly disagree for reverse scored items; 1 = agree for positively worded items or disagree for reverse scored items; and 0 = all other responses) and summed for each of the three time points. Scores ranged from 0–10, with higher scores indicating greater positivity toward SFH.

The second composite measure, an appropriateness index, was created by summing three items included post-training and post-implementation to assess staff beliefs about the appropriateness of SFH for 2-1-1 callers (the CFIR domain of outer setting). Items in this index included, The barriers in the lives of our callers will make it hard to implement SFH; I do not think SFH is appropriate for the kinds of callers we get; and I don’t think our callers are interested in SFH. Items were recoded as ordinal variables (2 = strongly disagree, 1 = disagree, and 0 = all other responses, including don’t know) and summed for each time point. Scores ranged from 0–6, with higher scores indicating greater perceived appropriateness for callers.

Post-implementation interviews with 25 staff from across sites were conducted by an evaluation team member and designed to enhance understanding of survey findings. Structured interview guide questions addressed the effects of local context on implementation, intervention adaptation, and the benefits and challenges of participation. Most questions were asked of all interviewed staff members, regardless of role. Up to 16 additional role-specific questions were added depending on the respondent’s role(s).

Data Analysis

We used SPSS 23.0 to analyze survey data. Analyses focus primarily on the post-implementation survey and items repeated at multiple time points. We conducted descriptive analyses and assessed change in attitudes/beliefs over time using repeated measures ANOVA (positivity index) and paired t-test (appropriateness index) (significance level p < .05).

Adobe Connect web conferencing software was used to conduct and audio record post-implementation interviews. The interviewer took notes using a standardized form based on CFIR constructs; although interviews were not transcribed, audio recordings were available for reference.22 The evaluation team discussed responses to determine major themes and triangulated interview responses with survey data.

Results

Surveys

Across sites, 126 staff completed at least one survey. Of these, 120 (95%) completed the pre-training survey, 101 (80%) completed the post-training survey, and 79 (63%) completed the post-implementation survey. Table 3 reports the number of survey respondents and responses to selected post-implementation items by site.

Table 3.

Responses from staff at the five 2-1-1 sites participating in the Smoke-Free Homes (SFH) project

	A	B	C	D	E
Number of participants
Completed pre-training (N = 120)	15	42	39	4	20
Completed post-training (N = 101)	11	35	36	4	15
Completed post-implementation (N = 79)	11	24	29	4	11
Completed all three surveys (N = 68)	10	22	25	2	9
Selected for qualitative interview (N = 25)	5	7	5	2	6
% agreement with selected post-implementation measures
Easy to adapt SFH to my 2-1-1	57%	50%	73%	75%	60%
Easy to make the SFH a permanent addition to our 2-1-1	36%	17%	35%	75%	36%
Received adequate training to deliver SFH	91%	88%	93%	100%	91%
Our SFH team had adequate staffing to carry out work	46%	75%	72%	75%	55%
SFH consistent with 2-1-1’s mission	82%	67%	79%	75%	100%
Individual skills a good fit for SFH tasks	82%	75%	90%	75%	100%
SFH tasks similar to normal 2-1-1 tasks	27%	46%	62%	75%	55%
Participants were satisfied	55%	67%	41%	50%	73%
Division of responsibilities for SFH clear to team members	82%	79%	93%	100%	73%

Percentages for post-implementation measures are % of people responding positively (agree/strongly agree, comfortable/very comfortable, easy/very easy). For the first item, people who responded don’t know/not applicable are treated as missing.

Table 4 provides results for all post-implementation survey items, as well as items repeated at multiple time points. Responses to intervention characteristic items showed a wide range. Post-implementation, 91% of staff reported receiving adequate training, 70% agreed the intervention had led to more smoke-free homes in their population, and 62% found it easy to adapt SFH to their 2-1-1. Only 32% believed it would be easy to make SFH a permanent addition to their 2-1-1.

Table 4.

Responses to Likert items from participants in the post-implementation survey (N = 79), organized by Consolidation Framework for Implementation Research (CFIR) domains

	Pre-training	Post-training	Post-implementation
Characteristics of intervention
Received adequate training to deliver SFH	—	—	91%
SFH led to more smoke-free homes in our 2-1-1 populationa	78%	88%	70%
SFH a better resource for callers with a smoker in the home than previous 2-1-1 offerings			62%
Easy to adapt to my 2-1-1b	—	—	62%
Without SFH funding, our 2-1-1 would not be able to offer it	—	—	57%
Delivering SFH took too much timea (disagree)	68%	77%	46%
Implementation was difficulta (disagree)	52%	66%	44%
Similar to other programs delivered at our 2-1-1	—	—	32%
Easy to make the SFH a permanent addition to our 2-1-1	—	—	32%
Characteristics of outer setting
SFH not appropriate for callers we getc (disagree)	—	76%	52%
Callers not interested in SFHc (disagree)	—	72%	35%
Barriers in lives of callers made it hard to implement SFHc (disagree)	—	41%	32%
Characteristics of inner setting
Communication among SFH team members excellent	—	—	87%
SFH seen by 2-1-1 leaders as important to mission	—	—	86%
Delivering SFH fit well with staff’s skills	—	—	81%
SFH consistent with 2-1-1’s mission	—	—	79%
2-1-1 management responded to concerns about SFH	—	—	72%
Did not have resources needed to be successful delivering SFHa (disagree)	82%	87%	71%
Our SFH team had adequate staffing to carry out work	—	—	67%
Research team responded adequately to technical concerns	—	—	58%
Enrollment goals realistica	60%	72%	51%
Characteristics of individuals
Individual skills a good fit for SFH tasks	—	—	85%
SFH staff took personal responsibility for program’s success	—	—	85%
Comfortable screening 2-1-1 callers for SFH eligibilityb	—	—	75%
SFH tasks similar to normal 2-1-1 tasks	—	—	52%
Comfortable recruiting participants into SFHb	—	—	72%
Comfortable delivering SFH coaching interventionb	—	—	71%
Comfortable contacting SFH participants for follow-upb	—	—	81%
Implementation process
Division of responsibilities for SFH clear to team members	—	—	85%
SFH team was highly coordinated	—	—	79%
Participants were satisfied	—	—	56%
Staff enjoyed working on SFH	—	—	51%
Easy to reach participants for coaching sessionsb	—	—	6%
Easy to reach participants for follow-up interviewsb	—	—	16%

^aItem included (with different verb tense) in pre-training and post-training surveys and included in the positivity index.

^bThese items were role-specific and thus one of the response options was don’t know/not applicable. Participants choosing that response are treated as missing.

^cItem included (with different verb tense) in post-training survey and included in the appropriateness index.

Data for pre-training and post-training items repeated at multiple time points is included for participants whose data were included in the repeated measures analyses (n = 68 and n = 71, respectively).

For positively worded items, percentage given is % of people responding positively (agree/strongly agree, comfortable/very comfortable, easy/very easy). For negatively worded items, as noted in the table, % is people responding negatively (disagree/strongly disagree).

Post-implementation, staff perceptions about the appropriateness of and caller interest in SFH (ie, outer setting) were mixed. About half of respondents (52%) disagreed with the statement that SFH was not appropriate for the callers they get, and 35% disagreed with the statement that callers were not interested in SFH.

In general, participants rated the characteristics of the inner setting (ie, their 2-1-1) quite favorably. Most respondents reported having excellent communication among SFH team members (87%). Some individual-level implementation characteristics (ie, 2-1-1 staff) were seen as a good fit for the SFH program. Most respondents (85%) thought their own skills were a good fit and reported that SFH staff took personal responsibility for the program’s success. Only half (52%) reported that SFH tasks were similar to their normal tasks at 2-1-1. Many felt comfortable delivering the coaching intervention (71%) or contacting SFH participants for follow-up (81%). Several aspects of implementation were highly rated. The majority of respondents reported that the division of responsibilities was clear among the team (85%) and thought their team was highly coordinated (79%). Very few staff, however, thought it was easy to reach participants for follow-up interviews (16%) or coaching sessions (6%).

Repeated measures analyses showed that staff positivity and views about participant appropriateness of SFH were significantly lower (p ≤ .001) after intervention delivery than prior to intervention delivery. The 68 participants with data for all three time points had mean positivity scores of 4.6 (SD = 2.7) pre-training, 4.8 (SD = 2.2) post-training, and 3.6 (SD = 1.9) post-implementation. The 71 participants with data post-training and post-implementation had mean appropriateness scores of 2.4 (SD = 1.5) post-training and 1.5 (SD = 1.6) post-implementation.

Interviews

Post-implementation interviews with staff (N = 25) provided context for survey results. Interviews afforded an in-depth look at the effects of local context, how the intervention was adapted, and challenges to implementation.

Fit Between Intervention and 2-1-1

Consistent with survey results, many respondents believed SFH was a good fit for their 2-1-1. One manager noted that their 2-1-1’s mission was bringing people and services together, and she saw SFH as a natural extension of that mission. Another manager, however, questioned whether a public health intervention was a good fit for a social services agency.

Effect of Local Context

The varied structures of the five sites affected the distribution of SFH responsibilities. One site had a designated staff member who carried out most SFH duties (and was not responsible for other 2-1-1 duties during the project period); staff at this site believed that approach worked well. Three sites used a team approach in which 2-1-1 staff combined work on SFH with their other duties for 2-1-1. Staff at these sites generally reported good levels of team cohesion among those implementing SFH. At one of these sites, over three dozen people were involved in prescreening and enrollment, but about 10 staff did the bulk of the coaching and follow-up calls. At the fifth site, a statewide 2-1-1 director worked with local site managers across nine sites to help them implement the program.

Consistent with the survey findings, most staff members believed they had adequate support from 2-1-1 leadership to deliver SFH. In some cases, leaders were seen as program “champions.” At one site, staff noted that one level of leadership was supportive of the program but another level of leadership was not, which led to mixed messages. A few staff members raised concerns about leadership that was “pushy,” especially when recruitment goals were not met.

Leadership used various means of encouraging staff participation in SFH, including food, recognition, and cash bonuses. Staff reported that they appreciated this external recognition but noted that they were motivated primarily by a desire to do their jobs well and not by external incentives.

Changes in Workflow and Duties

Several staff members reported that implementing SFH led to a change in their site’s work hierarchy. At one site, for example, a staff member’s promotion to SFH manager changed the reporting structure. Several participants reported discovering that as implementation progressed they needed dedicated time to work on SFH tasks, especially coaching and follow-up calls, which led to changes in workflow. In addition, typical metrics used to evaluate the performance of the 2-1-1 (eg, number of 2-1-1 calls answered, time callers were in queue) were not in line with conducting SFH calls. For example, the time spent conducting coaching and follow-up calls sometimes led to decreased performance indicators such as an increase in the number of 2-1-1 calls in the queue. In some cases, staff did not like being evaluated based on SFH performance metrics (eg, number of recruitment or coaching calls made) in addition to their usual performance metrics.

Several staff reported that although they were accustomed to fielding incoming calls as part of their 2-1-1 duties, proactively calling people (ie, for SFH coaching and follow-up) was a new role. Several mentioned that they enjoyed providing proactive assistance to callers as opposed to simply providing referrals, and they enjoyed hearing about participants’ success in establishing smoke-free homes. Others reported that some site staff were disengaged from SFH or did not feel comfortable asking participants questions about eligibility and enrolling them in the study, which was seen as pressuring callers. In response, some sites shifted staffing and assigned staff to SFH based on comfort level. Several staff members agreed that SFH worked best when it was delivered by staff members who were comfortable with their assigned tasks.

Adaptation

Most staff reported that training booster calls with research team staff and other sites were helpful. They liked hearing about how other sites overcame barriers and found effective solutions. To enhance participation in coaching calls, for example, one site began texting people reminders beforehand.

Many staff members wished that they had been able to modify timing of coaching and follow-up calls and adapt the SFH script. Call timing for all sites was determined by the study tracking tool; several staff members reported that they would have liked to be able to determine the timing of calls themselves based on prior attempts to reach participants. Staff reported experiencing conflict between wanting to relate to callers versus following a script they thought sounded impersonal. Several respondents reported making changes to script wording to make it more conversational or emphasize factors that might encourage callers to participate (eg, gift card incentives). Many staff also emphasized to participants or potential participants that SFH was not a tobacco cessation program; they thought doing so made it more acceptable to callers.

Resources

Staff time was a major resource needed for delivering SFH. Many staff members noted that reaching people for coaching and follow-up calls took longer than anticipated. Some staff experienced a conflict between conducting SFH calls and conducting their usual 2-1-1 duties. Staff members at one site reported that it would have been helpful to have more staff working on SFH.

Grant funding was used to purchase participant incentives ($10–25 gift cards). Staff believed that these incentives strengthened recruitment. Three sites used incentives from the beginning of the program; in two cases, incentives were added later to address recruitment challenges.

Sustainability

Most respondents (57%) in the post-implementation survey believed that their site would not be able to offer SFH without continued funding. In interviews, staff elaborated that they believed resources for participant incentives and staffing would be necessary to continue the program. A few sites were currently pursuing funding to continue offering SFH.

Overall Benefits and Challenges

Both survey responses and interviews showed that staff had mixed responses to participation in SFH. Although implementing SFH was more difficult than many staff anticipated, many said in interviews that, overall, they saw benefits to participating. Just over half of respondents reported that there were more rewards than challenges to participating in the program, and most of the remaining respondents reported that the rewards and challenges were about equal. Most staff interviewed said that they were more likely to participate in future research after participating in SFH. One staff member reported that participating had opened her eyes and opened the doors to future opportunities to take on public health projects at 2-1-1.

Discussion

We used CFIR constructs to conduct a mixed-methods evaluation of the implementation of an evidence-based health intervention at five US sites. Incorporating the SFH program into 2-1-1 call centers offers a unique opportunity to reach low-income smokers who may struggle to prioritize health. Evaluation results can inform future efforts to increase the dissemination and scalability of the SFH intervention to 2-1-1 sites by providing information about the program’s alignment with 2-1-1’s mission, as well as the acceptability and feasibility of delivery.²³

A nuanced picture emerges. Overall, staff who responded to the surveys viewed inner setting (that is, individual 2-1-1 sites) favorably. In interviews, staff described a generally positive implementation climate, including leadership support. These results are consistent with quantitative data showing that most respondents thought leadership saw SFH as important to their 2-1-1’s mission (86%) and found SFH consistent with their 2-1-1’s mission (79%). Although characteristics of the outer setting (eg, characteristics of 2-1-1 callers) were perceived as less favorable to SFH implementation, interview results suggest that participating in SFH led many 2-1-1 staff to be more likely to participate in such intervention partnerships in the future. Participation in SFH may have empowered them to develop new skills and think more broadly about how to meet health needs in their communities.

Findings were more variable for CFIR domains related to characteristics of the SFH intervention, implementation staff, and implementation process. A large majority of respondents (91%) believed that their SFH training had been adequate. Staff believed SFH was effective in their local contexts; 70% thought that the program led to more smoke-free homes in their target population. Overall, 85% of respondents believed that their own skills were a good fit for SFH, and many were comfortable delivering the coaching intervention or contacting participants for follow-up. Very few believed it was easy to reach participants for coaching or follow-up. Only 32% of respondents believed it would be easy to make SFH a permanent addition to their 2-1-1. These results suggest that although the goal of SFH may be a good fit with 2-1-1’s mission, the approach (eg, particular duties required, training received) may need modification in order for staff to feel comfortable with intervention delivery.

Repeated measures data revealed a decline from pre-intervention to post-intervention in composite measures of positivity and appropriateness. This shift may reflect 2-1-1 staff’s development of a better understanding of SFH implementation demands after direct involvement. Interviews revealed frustrations about parts of the intervention delivery process that may have been driving these results. Staff expressed concerns that the skills needed to deliver SFH differed from those needed to perform their 2-1-1 duties (eg, responding to callers’ needs vs. proactively recruiting them), and they wished they had been able to do more to adapt the SFH script and modify timing of coaching and follow-up calls and number of required call attempts. To address these concerns, some staff reported making minor adaptations to content by altering how they used the SFH database and shifting modes of contact (ie, using texting).

Staff also expressed concerns about intervention sustainability. Although they believed participants were satisfied with the intervention, they believed incentives (ie, gift cards) were helpful in encouraging participation. This finding suggests that, even with brief interventions such as SFH, it may be necessary to offer compensation to motivate potential participants to enroll. In this study, incentives were purchased using sites’ grant funding; if SFH were to continue to be offered, sites would need additional funding for incentives (as well as other expenses such as staff training and staff time).

Scalability of interventions can be informed by studying pragmatic implementation beyond a carefully controlled research environment. It is important for practitioners to assess whether a research-tested intervention’s findings apply to a particular local setting, population, or resource. Results suggest that when delivering a new intervention at an agency, it is important to have support from leadership at multiple levels, not just a single “champion.” It is also crucial to consider how existing workflow and performance metrics may affect intervention delivery. Where possible, implementation of new efforts should be compatible with existing performance measures and incentives; if not, implementation performance measures should be adjusted. Skills and interests of staff should also be evaluated to ensure a good fit with intervention requirements.

Findings underscore the importance of balancing intervention fidelity (eg, adhering to a script) with flexibility and adaptation. Additional training could address concerns about sounding scripted or impersonal by specifying which deviations from the script are acceptable while still maintaining intervention fidelity. Incorporating contact methods such as texting, as well as allowing flexibility in the timing and number of call attempts, may be helpful in addressing the challenges of contacting low-income populations.

Strengths of this study include a mixed-methods, longitudinal approach to examine implementation and scalability, including online surveys administered at three time points. Interviews with staff allowed in-depth exploration of SFH implementation at five sites.

There are also limitations. Results may not generalize to 2-1-1s serving other geographic areas or to social service agencies with different service models or staffing structures. Sites were chosen through a rigorous selection process, and their leadership and/or staff may be exceptionally motivated to participate in health interventions. Another limitation is that all data come from staff self-report. Although staff were told data would be reported only in aggregate, social desirability bias may have influenced responses, especially in interviews. Despite multiple reminders from evaluation and site staff, not all staff members responded to all surveys; only 62% of staff who completed one of the pre-implementation surveys completed the post-implementation survey. Most non-response was due to staff turnover, which is not uncommon in social service agencies; if turnover or non-response were differential in some systematic way (eg, staff participating in the final interview were more skilled than those no longer at 2-1-1, or dissatisfied staff were more likely to respond), it could influence the pattern of responses. These analyses are primarily descriptive; multilevel analyses with more sites could determine which site characteristics best predict program reach, adoption, and health outcomes among participants. Cost-benefit analyses could also provide further insight into the potential scalability of SFH.

This work examined the implementation of an evidence-based tobacco control intervention at multiple delivery sites of a social service agency. We found that SFH was generally acceptable and feasible among sites, but there were also challenges around adaptability and sustainability. Future implementation of SFH and other tobacco control programs should address barriers identified in this research by securing sustainable funding, providing dedicated staff time, and ensuring that programs fit with staff workflow. Addressing such challenges will aid efforts to scale up evidence-based tobacco control interventions to social service agencies.

Funding

This work is supported by the National Cancer Institute at the National Institutes of Health through the State and Community Tobacco Control Research Initiative (grant number U01 CA154282).

Declaration of Interests

None declared.