A mixed-methods study to understand the impact of practitioner and organizational factors on fidelity of a child maltreatment prevention intervention in community-based settings

Abstract

It has been well-documented that the degree to which interventions are implemented with fidelity in typical service settings has varied. Frequently, interventions are developed and tested in highly controlled or early adopter settings. Less attention has been given to what implementation looks like in usual care, and which factors promote practitioners' ability to implement with fidelity. Individuals and organizations implementing interventions in the real world receive varying levels of external supports and may apply a new intervention unaided. The purpose of this mixed-methods study was to explore factors that support implementation as intended in local community agencies. In the quantitative phase of this study, 32 case planners implementing Family Connections (FC), a child maltreatment preventive intervention, completed a survey about their perceptions of practitioner and organizational factors related to fidelity. The survey data were connected to case-level fidelity scores to understand the relationship between perceptions and fidelity. The qualitative phase of this study involved further exploration with nine case planner interviews and two separate focus groups with supervisors and agency leadership. The results of this study suggest that supervision is a key contributor to a practitioner's ability to implement an intervention in usual care. The quantitative and qualitative results suggest supervision, including supervisors’ perseverance, proactiveness, knowledge, availability, and skill reinforcement are important components of enhancing a practitioners' ability to learn and use FC. The quantitative results suggest that the level of education was positively associated with fidelity and perceptions of the intervention's limitations may be negatively related to implementation. Additional components that influence implementation for future research emerged from the qualitative phase related to system expectations and policies, individual practitioner attributes, and characteristics of the intervention.

Plain Language Abstract

This mixed-methods study sought to understand the impact of practitioner and organizational factors on fidelity of a child maltreatment prevention intervention in community-based settings. The study first asked case planners about their perceptions of practitioner and organizational factors related to fidelity through an online survey. This survey was connected to case-level fidelity scores to understand the relationship between perceptions and fidelity. The qualitative phase of this study involved further exploration with nine case planner interviews and two separate focus groups with supervisors and agency leadership. The results of this study suggest that supervision is a key contributor to a practitioner’s ability to implement a maltreatment prevention intervention. Both methods of the study suggest that various aspects of supervision, including supervisors’ perseverance, proactiveness, knowledge, availability, and skill reinforcement are important components of enhancing a practitioner's ability to learn and use the intervention. Additional components that influence the implementation for future research emerged from the qualitative phase related to system expectations and policies, individual practitioner attributes, and characteristics of the intervention.

Background

There is a consensus that the selection of an evidence-based program does not equate to its effective implementation in the real world, and that ongoing monitoring and support for the intervention is required (Mildon & Shlonsky, 2011). Fidelity is one way to monitor an intervention's implementation and is defined as the degree to which an intervention is implemented as intended by the program developers (Dusenbury et al., 2003). However, there is a limited understanding of intervention fidelity because it is poorly articulated in the literature in usual care, and when it is, there are significant limitations in interpreting the results such as limited information on reliability and validity of measures (Mowbray et al., 2003). Also lacking is what contributed or impacted intervention fidelity in community-based settings.

A variety of factors have been conceptualized to influence fidelity, but many go untested. Carroll et al.'s (2007) conceptual framework identified that the degree of complexity and operationalization of an intervention, implementation support strategies, and quality of delivery influence fidelity. At the organizational level, leadership, workflow, and reinforcement have been positively associated with higher rates of fidelity (Torrey et al., 2012). Other studies suggest that ongoing fidelity checks and consultation result in high fidelity (Self-Brown et al., 2012), which are often carried out by supervisors. However, the results of supervision effects on fidelity have been mixed. For example, consultation, coaching, and supervision contributed to improvements in practitioners’ skills and fidelity (Edmunds et al., 2014; Nadeem et al., 2013); yet a review of supervision in child welfare highlighted weaknesses in supervision, calling into question how supervision influences fidelity (Carpenter et al., 2013). Another review suggests that higher fidelity was related to supervisory practices and the use of continuous quality improvement strategies at the organization level; while results were mixed at the practitioner level (Bartley et al., 2017). It is possible that high fidelity requires a constellation of support. For example, when strategies such as manual, active training elements (e.g., role-play), and organizational supports (e.g., web-based support, group consultation) were enacted at all levels, therapists were more likely to achieve behavior change and new skills with fidelity (Beidas & Kendall, 2010).

In order to understand fidelity and its contributors, we must study interventions in typical service settings. This allows for an increased understanding of the contextual factors that influence the implementation of interventions. Family Connections (FC) is a child maltreatment preventive intervention that was developed based on the principles of prevention science to provide weekly contact with a family in their home or neighborhood to meet the basic needs of their children, reduce the risk of child maltreatment, and increase family functioning (DePanfilis & Dubowitz, 2005). FC has been shown to be effective in improving caregiver protective factors, decreasing risk factors, and improving child well-being (James Bell Associates, 2011). Replication studies of FC's implementation showed that higher fidelity led to greater significant outcomes over time (DePanfilis, 2015). This study's purpose was to understand contributors to fidelity through a collaborative of community-based organizations replicating FC.

Method

This study employed an explanatory sequential mixed-methods design (Creswell, 2014) to understand the relationship between practitioner and organizational factors and fidelity. This design is ideal for understanding complex situations (Creswell & Plano Clark, 2007) and implementation research (Aarons et al., 2011) of dynamic environments. The study aimed to examine the relationship between organizational factors (social supports, quality assurance strategies, climate, and workload) and FC fidelity scores; to examine the relationship between individual practitioner factors (age, years of experience, degree, length at agency, attitudes toward evidence based programs (EBPs), and self-efficacy) and FC fidelity scores; to use the results of quantitative components to further explore staff perceptions of organizational and individual factors that are beneficial to worker fidelity of FC; and to integrate quantitative and qualitative research findings in order to more fully understand factors related to fidelity in a child maltreatment prevention context.

Procedures

The study included eight community-based agencies replicating FC who were part of an effort to integrate evidence into a metropolitan city's preventive child welfare services. Three implementation science frameworks were used to conceptualize the primary factors associated with fidelity (Aarons et al., 2011; Durlak & DuPre, 2008; Meyers et al., 2012). In the first phase of the study, quantitative methods were used to gather survey data and case-level fidelity scores. Based on the quantitative findings, the second phase included a qualitative phenomenological analysis (Creswell, 2014), in which semistructured individual interviews were conducted with case planners (i.e., practitioners working with families), as well as two focus groups conducted with supervisors and agency leadership.

Participants

Quantitative

The survey included a convenience sample of case planners from the eight FC agencies. All case planners currently carrying cases were invited to participate in the study. Table 1 contains descriptive information on the case planners who completed the survey. A total of 42 of 60 possible respondents completed the survey, for an initial response rate of 70.00% from seven of the eight implementing agencies. Five respondents identified themselves as noncaseworkers (e.g., supervisors or other) and were therefore excluded from the sample, for a final response rate of 67.27%.

Table 1.

Survey participant characteristics.

Characteristic	Full sample, N	M (SD) or %	Fidelity scores, N	M (SD) or %	T or χ2
Gender
Female	35	94.7	30	93.75	0.37
Age	32	34.40 (9.48)	26	34.04 (9.46)	−0.45
Race					2.67
White (non-Hispanic)	3	7.9	3	9.4
Black/African American	17	44.7	15	46.9
Native American/Alaska	1	2.6	1	3.1
Asian/Pacific Islander	1	2.6	1	3.1
Hispanic/Latino	15	39.5	11	34.37
Other	1	2.6	1	3.1
Highest level of education					1.16
BSW	4	10.5	3	9.4
Other BA/BS	24	63.2	21	65.6
MSW	3	7.9	2	6.2
Other MA/MS	7	18.4	6	18.8
Length in current position	37	30.95 (28.49)	32	32.25 (28.81)	0.65
Length at agency	37	55.16 (69.58)	32	59.31 (74.03)	0.85
Length in child welfare	36	79.95 (70.75)	31	85.03 (74.68)	0.99
Number of families	36	7.03 (1.9)	32	7.41 (1.37)	3.51**

*p ≤ .05, **p ≤ .01, ***p ≤ .001 (two-tailed).

Qualitative

The qualitative portion of the study included individual interviews with case planners and two separate focus groups with supervisors and leaders. Table 2 summarizes case planners’ characteristics based on information gathered from the survey.

Table 2.

Qualitative interview participant characteristics (n  =  9).

Characteristic	N	% or M (SD) or %
Gender
Female	7	77.8%
Age	5	31.2 (6.91)
Race
White (non-Hispanic)	1	11.13
Black/African American	4	44.41
Native American/Alaska Native	1	11.12
Hispanic/Latino	3	33.34
Highest level of education
BSW	1	11.11
Other BA/BS	8	88.93
Length in current position	9	38.11 (34.45)
Length at agency	9	68.11 (69.44)
Length in child welfare	9	86.22 (72.68)
Number of families on caseload	9	7.56 (1.91)
Average fidelity score	9	78.29 (11.59)

The second qualitative component was two separate focus groups with agency supervisors and leaders. All supervisors currently overseeing FC implementation were invited to participate and four (50%) of the eight supervisors participated. Supervisors did not complete the case planner survey; therefore, only agencies that they currently worked with were identified.

As for the focus group with FC leaders, 6 of the available 14 leaders participated. These leaders came from five different agencies and were a mix of large (>20 case planners) and small (<10 case planners) FC programs.

Measures, data collection, and analysis procedures

Quantitative

The electronic FC Worker Survey was designed based on the Tailored Design Method (Dillman, 2009) to assess the independent practitioner and organizational variables. Practitioner data included demographics, education and experience, attitudes toward evidence-based practice using the Evidence-Based Practice Attitudes Scale–50 (EBPAS-50; Aarons, Cafri, et al., 2012), and self-efficacy using the Self-Efficacy Scale for Social Workers (Pedrazza et al., 2013). Organization characteristics included social supports through the Implementation Leadership Scale (Aarons et al., 2014); quality assurance strategies through three subscales of the EBPAS-50 (Aarons, Cafri, et al., 2012), monitoring, organizational support, and feedback; organizational climate was measured through the 18-item Implementation Climate Scale (Ehrhart et al., 2014) and workload, which was measured by the number of families a worker currently served. All study procedures were approved by the University's Institutional Review Board, and all participants completed informed consent forms.

Fidelity to FC

Fidelity was analyzed through data from routine biannual case reviews. Reviews include an agency self-assessment and an independent case review conducted by FC expert consultants. Using case reviews to measure fidelity is a practical and efficient source for understanding implementation in typical service settings (Kaye & Osteen, 2011).

In the routine fidelity data collection process, each participating FC organization submitted a list of opened and closed cases during the review period. A random numbers generator was used to select a sample of opened and closed cases from each FC program. A team of expert FC consultants, who had gone through training and supervised the implementation of the fidelity review protocol, went on-site to review this random sample. The fidelity case review instrument included 77 items that measured the degree to which case planners implement FC practice components of Intake/Screening, Outreach and Engagement, Concrete/Emergency Services, Comprehensive Family Assessment, Outcome-Driven Service Plan with SMART Goals, Change Focused Intervention, Service Plan Evaluation/Progress Assessment, and Case Closure. These domains are consistent with a previous fidelity measure implemented in a national cross-site evaluation of the replication of FC (James Bell Associates, 2011). The majority of items are rated as yes  =  1, no  =  0, or not applicable. There are also items included in the assessment with ordinal ratings of sufficient  =  1, partially sufficient  =  0.5, and insufficient  =  0. Some items in the review instrument are not applicable to all cases; therefore, proportional scores were calculated for each component section (Intake/Screening, Outreach and Engagement, Concrete/Emergency Services, etc.). For the purpose of this study, the average fidelity score ranged from 70% to 85% range, with above average parameter at >85% and below average <85%. This was based on the distribution of the sample; the average fidelity score was 77.71 (SD  =  9.60) with a range of 54.60–94.9. Acceptable fidelity scores are >70% (James Bell Associates, 2011). Each case review resulted in an average fidelity score across the FC core components for a unique case score. Not all survey respondents had fidelity scores (because they had not yet had cases reviewed through the fidelity review process), and therefore, could not be included in the primary analyses. Independent samples t tests for categorical variables and Pearson chi-square tests for continuous variables were used to assess any differences between the full sample (n  =  37) and sample with fidelity scores (n  =  32) and are reported in Table 1.

Assessment of the practitioner and organizational factors’ relationship with fidelity scores was done through two statistical approaches. Assumptions were assessed for all predictors at each level of the Growth Linear Mixture Model (GLMM; Tabachnick and Fidell, 2007). Specifically, heteroscedasticity, normality of level one residuals, homogeneity of variance across clusters, normality of residuals at higher levels, and multicollinearity of predictors were assessed through histograms, and comparing predictor and actual values. To understand organizational variables, a GLMM with bootstrapping was used to assess the relationship between the organizational variables and case planner fidelity scores. The multilevel model was appropriate given the nested fidelity case data within workers (Bickel, 2007). This preliminary model detected a large degree of variability within the dependent variable (case fidelity scores) suggesting heteroskedasticity, which was further supported by the results of the assumption testing. Heteroskedasticity occurs when the variability among the dependent variable differs for grouping variables, but it is not fatal to the analysis (Tabachnick & Fidell, 2007). Therefore, a series of multilevel models with bootstrapped standard errors were used to control for the inconsistent predictive utility across values of the dependent variable. The first model included all organizational factors, which was based on the original research question. Then, considering model fit and the conceptual domains of the variables, model 2 included factors associated with supervision, and model 3 included factors associated with organizational factors to disaggregate some of the model one results. For individual factors, an Ordinary Least Squares regression was used with degree (Masters vs. Bachelors, with Masters as reference category), length at agency (in months), attitudes toward EBPs, and self-efficacy as the independent variables and average fidelity score as the dependent variable.

Qualitative

Following quantitative data collection, semistructured 30–40 min interviews were conducted with case planners. Additionally, two focus groups were conducted with a group of supervisors and a group of leaders. Participants were identified through their agency and invited to participate. A protocol based on phase 1 results was used in the interviews and focus groups and included nine open-ended questions with optional probes that included areas of inquiry related to significant individual and organizational variables related to fidelity included: perceived limitations of FC, level of education, proactiveness, knowledge, and perseverance of supervisor. Additionally, factors that were not found to be significant but had narrow confidence intervals were also explored qualitatively related to job security of EBP, support self-efficacy, attitudes of feedback, attitudes of monitoring, education support of EBP, and number of cases.

Interviews and focus groups were audiotaped and transcribed verbatim and analyzed in Dedoose, with the first author coding all data and a research assistant coding half of the data. The coders met on three occasions to discuss the coding process, review differences in coding, and resolve discrepancies. A two-cycle coding process (Saldaña, 2013Saldaña, 2013) was used to analyze individual interviews and focus group data and identify the individual and organizational factors identified by case planners. The first cycle used a combination of evaluation and In Vivo coding and was done separately for a group of interviewees. The second-order coding involved pattern coding, which allows for grouping and summarizing data into succinct themes and constructs (Miles, Huberman, & Saldaña, 2014).

Mixed methods

The last phase of analysis involved the integration of data to understand how the results of the second phase further explained the quantitative results (Creswell, 2014). To do this, an average fidelity score descriptor (i.e., above average, average, and below average) was added to each case planner interview file in Dedoose. Then, we examined qualitative codes by the different fidelity thresholds to see whether case planners who scored above or below average spoke more or less about particular factors.

Results

Quantitative

Descriptive statistics were used to analyze average organizational factor scores and overall average fidelity scores across FC agencies (see Table 3) and were inconclusive as to whether agencies with higher and more positively perceived organizational factors compared to other FC agencies had higher overall fidelity scores. In examining the agencies with above average fidelity scores (i.e., agencies two, three, and eight), there appears to be a range of organizational factor average scores. Agency eight had higher and more positively perceived organizational factors overall, whereas agencies two and three had a range of low and high scores across organizational factors. Additionally, although agency two had a higher overall average, this is based on only three workers, and perceptions of organizational factors are generally lower than agency three and eight. The sample sizes were small for all agencies except one (agency three) and despite these limited data suggest other factors beyond organizational supports may affect fidelity. Therefore, a series of multilevel models were used to assess whether particular organizational factors were associated with higher case fidelity scores. Residual interclass correlations for each model are provided in Table 4. See the Fidelity to FC section for a description of fidelity parameters.

Table 3.

Average organizational factor and fidelity scores across FC agencies.

Parameter	Overall and FC agency mean scores
	Overall M (SD) (n = 32)	1 (n = 1)	2 (n = 3)	3 (n = 15)	4 (n = 4)	5 (n = 3)	6 (n = 2)	8 (n = 4)
Fidelity	77.71 (9.6)	72.22	82.08 (0.59)	78.88 (10.97)	75.00 (4.35)	64.60 (8.66)	74.14 (2.14)	85.74 (2.78)
Proactive sup1	2.43 (1.15)	2.3	1.88 (0.69)	2.44 (1.21)	2.83 (0.88)	1.33 (1.53)	2.16 (1.65)	3.42 (0.50)
Knowledgeable sup1	2.91 (1.04)	4.0	2.0 (0.88)	2.93 (0.87)	3.58 (0.83)	1.44 (1.39)	3.50 (0.71)	3.42 (0.42)
Supportive sup1	3.0 (1.12)	4.0	2.55 (0.77)	2.98 (1.17)	3.66 (0.66)	1.44 (1.50)	3.50 (0.71)	3.42 (0.42)
Perseverant sup1	2.84 (1.16)	3.33	2.33 (0.77)	2.88 (1.14)	3.50 (1.0)	0.77 (0.69)	3.50 (0.71)	3.50 (0.43)
Sup leadership1	2.79 (1.05)	3.42	2.19 (0.60)	2.81 (1.04)	3.40 (0.83)	1.25 (1.26)	3.16 (0.94)	3.44 (0.43)
Org support2	2.80 (1.05)	3.66	2.22 (1.35)	2.73 (0.98)	3.42 (1.16)	2.22 (1.57)	3.50 (0.71)	2.75 (0.87)
Feedback2	3.10 (0.99)	3.33	2.22 (1.39)	3.09 (0.99)	3.33 (1.33)	3.22 (0.84)	3.50 (0.71)	3.25 (0.96)
Monitoring2	1.11 (1.11)	1.25	2.17 (1.01)	1.30 (1.15)	0.50 (0.84)	0.75 (0.90)	0.13 (0.18)	1.0 (1.41)
Openness to EBP2	2.10 (1.00)	.49	1.87 (0.84)	2.14 (0.87)	1.95 (1.13)	0.90 (1.05)	3.11 (0.85)	3.05 (0.41)
Recognition of EBP3	1.0 (1.0)	1.0	1.22 (0.84)	0.70 (0.77)	1.17 (1.35)	0.66 (0.57)	1.15 (1.15)	2.83 (1.73)
Ed Support for EBP3	2.40 (1.08)	3.33	1.89 (0.69)	2.08 (1.15)	2.92 (0.83)	1.77 (0.51)	3.0 (1.42)	3.38 (0.95)
Selection of EBP3	1.08 (1.15)	0	0.67 (0.57)	0.99 (1.02)	0.92 (1.03)	0.22 (0.38)	2.33 (2.36)	2.16 (1.29)
Reward for EBP3	0.71 (0.86)	0	0.44 (0.51)	0.64 (0.66)	0.50 (0.64)	0.00 (0.0)	0.54 (0.76)	2.13 (1.11)
Focus on EBP3	3.0 (0.88)	4.0	1.55 (0.69)	2.74 (0.71)	3.66 (0.38)	2.66 (0.57)	3.83 (0.24)	3.91 (0.16)
Imp climate overall3	1.73 (0.78)	1.47	1.27 (0.49)	1.55 (0.57)	3.66 (0.38)	1.04 (0.28)	2.33 (1.13)	2.91 (0.86)
Number of cases	7.71 (1.36)	3	8.33 (0.57)	7.47 (0.83)	7.75 (0.50)	8.0 (2.0)	5.0 (1.41)	8.0 (0)

Note. FC = Family Connections; sup = Supervisor; Org = Organizational; Imp = Implementation.

1  =  Implementation Leadership Subscale (ILS) (0-4).

2  =  Evidence-Based Practice Attitudes Scale (EBPAS) (0-4).

3  =  Implementation Climate Scale (ICS) (0-4).

Table 4.

Residual interclass correlations for tested models.

Model	ICC	SE	CI
One	8.61e-23	0	[8.61e-23, 8.61e-23]
Two	0.19	0.08	[0.08, 0.39]
Three	0.16	0.07	[0.05, 0.35]

As shown in Table 5, only factors related to supervision were significantly associated with case fidelity scores across models one and two compared to other organizational factors. Specifically, in the first model with all organizational factors, higher fidelity scores were related to more positive perceptions of supervisor proactiveness (b  =  7.44, p  =  .03) and more negative perceptions of supervisor knowledge (b  =  −11.6, p  =  .01). In the second model of supervisor-related factors only, perceptions of supervisor proactiveness were again positively related to higher fidelity scores (b  =  5.58, p  =  .005), as well as perceptions of supervisor perseverance (b  =  5.23, p  =  .03), whereas supervisor knowledge was negatively related to higher fidelity scores (b  =  −7.17, p  =  .005). No factors were significant in the third model of organizational factors.

Table 5.

Fixed effects estimates, bootstrapped SE's, and CIs for models predicting worker fidelity (level 1  =  112).

Parameter	Model 1		Model 2		Model 3
	b	CI	b	CI	b	CI
Proactive supervisor1	7.44 (3.50)*	[0.57, 14.30]	5.58 (2.15)**	[1.67, 9.49]
Knowledgeable supervisor1	−11.6 (4.55)*	[−20.52, −2.68]	−7.17 (2.04)**	[−11.17, −3.16]
Supportive supervisor1	−4.61 (7.11)	[−18.55, 9.33]	0.50 (3.01)	[−5.39, 6.39]
Perseverant supervisor1	8.89 (5.48)	[−1.86, 19.64]	5.24 (2.55)*	[0.25, 10.23]
Attitudes of feedback2	−3.06 (3.27)	[−9.47, 3.35]	−0.39 (1.13)	[−3.00, 2.20]
Attitudes of org support2	5.57 (4.88)	[−4.0, 15.14]			3.10 (1.70)	[−0.23, 6.45]
Attitudes of monitoring2	0.88 (2.35)	[−3.74, 5.50]			−0.40 (1.20)	[−2.77, 1.97]
Selection for openness2	1.11 (2.29)	[−3.36, 5.6]			2.35 (2.11)	[−1.79, 6.48]
Recognition of EBP3	2.41 (2.65)	[−2.79, 7.60]			1.80 (2.26)	[−2.65, 6.26]
Educational support for EBP3	2.13 (2.68)	[−3.12, 7.39]			0.71 (1.52)	[−2.27, 3.69]
Selection of EBP3	−2.51 (2.89)	[−8.19, 3.15]			0.08 (2.14)	[−4.11, 4.29]
Reward for EBP3	−0.65 (3.41)	[−7.34, 6.02]			1.49 (3.44)	[−5.24, 8.25]
Focus on EBP3	0.98 (3.34)	[−5.56, 7.54]			−1.14 (1.73)	[−4.54, 2.51]
Number of cases	−2.86 (2.61)	[−7.99, 2.27]			−0.70 (1.60)	[−3.83, 2.42]

*p < .05, **p < .01, ***p < .001 (two-tailed).

1  =  Implementation Leadership Subscale (ILS) (0-4).

2  =  Evidence-Based Practice Attitudes Scale (EBPAS) (0-4).

3  =  Implementation Climate Scale (ICS) (0-4).

As shown in Table 6, only two individual-level variables were associated with fidelity scores; relative to those with master's degrees, those with bachelor’s degrees scored lower on average (b  =  −4.85, p  =  .05). Additionally, perceptions of the limitations of implementing FC were negatively associated with fidelity scores (b  =  −5.08, p  =  .02). On average, individuals with a higher perceived value of FC had higher fidelity scores.

Table 6.

OLS regression results of individual factors and average fidelity scores.

Variable	b	SE b	β	t	p	CI
Level of education—Bachelors	−4.85	2.22	−.46	−2.18	.05*	[−9.7, 0.04]
Level of education—Masters
Fit of EBP2	−0.99	1.94	−.11	−0.51	.62	[−5.25, 3.27]
Limitation of EBP2	−5.08	1.80	−.54	−2.82	.02	[−9.03, −1.22]
Balance of EBP2	3.88	2.24	.359	1.73	.11	[−1.05, 8.81]
Burden of EBP2	−2.92	2.65	−.24	−1.10	.30	[−8.76, 2.92]
Job security of EBP2	2.06	1.53	.26	1.34	.07	[−1.32, 5.44]
Emotional regulation4 self-efficacy4	4.38	2.18	.52	2.01	.07	[−0.43, 9.18]
Procedural self-efficacy4	−5.72	2.69	−.58	−2.12	.06	[−11.64, 0.21]
Support self-efficacy4	−0.06	1.35	−.01	−0.05	.96	[−3.02, 2.90]
Length at agency (months)	−2.46	1.27	−.42	−1.93	.08	[−5.26, 0.34]

R²  =  .699.

*p < .05.

2  =  Evidence-Based Practice Attitudes Scale (EBPAS) (0-4).

4  =  Social Worker Self-Efficacy (SE) Scale (1-7).

Qualitative

The qualitative research component examined case planners’, supervisors’, and leaders’ perspectives on the individual and organizational factors that are related to FC fidelity. Table 7 provides an overview of themes, similarities, and differences across the respondent groups. Interviews with case planners suggested a variety of characteristics that are important in a supervisor: they know the intervention, they are available and supportive, and they provide case consultation and brainstorm with case planners experiencing barriers. For example, one case planner shared “What they do is when we are in supervision and if we’re stuck with a case, instead of giving us the answer, they try to guide us to how to help the client.” Additionally, interviews with case planners suggested that coworkers have an important role in providing ongoing support for their work. Case planners suggested that both relational and technical attributes are required for effective FC work. Case planners reported mixed responses regarding whether FC enhanced their work with families, seemingly dependent on the family's situation and immediate needs. For example, six of the nine case planners discussed the tension between family needs and other expectations and time frames from the funding agency. These could be compounded by a lack of available resources (i.e., housing or material supports) to aid families with immediate needs.

Table 7.

Summary of qualitative results across respondents.

Group	Organizational	Practitioner	Intervention
Case planners	Supervision Coworkers Training Leadership Child Welfare Policies	Relational attributes—empathy, persistence, and passion Technical attributes—organization, timeliness, and physical strength FC enhances or somewhat enhances work with families	Theory and approach for working with families is helpful, FC provides a strong framework for their work, timeframes can be challenging in working with certain families
Supervisors	Resources Support for Supervisors Parallel Process Supervisor Knowledge Training Leadership Child Welfare Policies	Technical attributes—organization, timeliness, critical thinking, ability to receive feedback	Particular aspects of the intervention require ongoing support in order to achieve fidelity—interpreting the CASI, creating service plans, and developing SMART goals. FC enhances work with families.
Leaders	Resources Supervision Training Child Welfare Policies	Technical attributes—Organization, discipline, determined, time management Relational attributes—empathy, passion	Fidelity assessment process—required documentation and timeframes. FC “elevated the practice” FC enhances skill of workers, provides a framework, and enhances work with families.

Note. FC = Family Connections.

The supervisor focus group highlighted the importance of providing ongoing support through training and coaching to not only case planners but also supervisors in order to promote case planner FC fidelity. Supervisors reported on the technical attributes of case planners’ individual characteristics and emphasized the importance of the ability of a case planner to be open to feedback as a critical attribute related to FC fidelity. They also underscored that particular components of the FC model required ongoing support for case planners to achieve fidelity.

Leaders emphasized the importance of access to consultants that know the model and are available to provide ongoing support for practice improvement. Leaders also described how supervision was critical for ensuring FC fidelity through case planner skill development and application. They agreed that the fidelity assessment process was cumbersome and too frequent in order to apply recommendations and findings to practice and improve fidelity results. Leaders felt strongly that FC had “elevated” the practice of case planners’ work with families in preventive services. As one leader described:

Family Connections gives the workers something that they didn’t have before. In my mind, it’s structure. It brings the structure and an approach to working with families. Prior to that was case management and each agency would determine what the case management component stages are. This does have case management built into it, but it's very structured.

Mixed methods

For case planners with below average fidelity scores, organizational factors related to leadership, consistency of practice expectations from supervisors, caseloads, and paperwork requirements were major themes that emerged. As a case planner with a below average score noted:

I think it's more of…not understanding and supporting the workers in regard to how many cases they’re giving and how many visits have to be done, and not recognizing that with all the paperwork that needs to be done for Family Connections, giving a little bit more leeway instead of sticking to the guidelines so strictly.

This case planner explained that challenges were raised by both leadership and supervisor expectation regarding FC implementation. In terms of individual case planner characteristics, case planners with below average fidelity scores noted that strong writing abilities and having the creativity to create strategies with families were important characteristics. Case planners with below average fidelity scores also had mixed views on the degree to which FC enhanced their work with families.

Case planners with above average fidelity scores noted themes related to supervision, specifically related to how their supervisors collaborate and brainstorm in coordination with case planners on solutions for case challenges. They also emphasized specific techniques such as role-plays and discussion of goals in supervisory sessions. Case planners with above average fidelity scores emphasized the availability of their supervisor, both in terms of consultation ad hoc, but also in discussing concerns, barriers, or additional needs. No individual factors emerged as unique to case planners with higher average fidelity scores.

There was convergence in the qualitative and quantitative results regarding supervisor characteristics of proactiveness, perseverance, attitudes toward feedback and monitoring, educational supports, and number of cases. Results from the interviews with case planners suggest that when a supervisor was available, supportive, and helped to attend to barriers, case planners felt as though they could implement FC with fidelity. Case planners spoke frequently about how the support of the supervisor to help them brainstorm and persevere through challenges was critical. Case planners who had more positive perspectives on feedback also shared its usefulness in improving the quality of FC practice with families. This was corroborated by the supervisor focus group, which identified the importance of case planners’ receptivity to feedback. Although not statistically significant in quantitative models, across case planner interviews and focus groups, educational support through training and a manual and resources for case planner documentation were referenced as helpful, but insufficient in ensuring fidelity to FC. Across qualitative data, the number of cases was seen as an important factor influencing fidelity. Case planner interviews and supervisor focus group information suggested that the addition of just one case, or having multiple cases with competing deadlines, can hinder a case planner's ability to achieve fidelity related to documentation and required time frames.

In terms of individual factors, case planners with below average fidelity scores reported mixed perspectives regarding FC's ability to enhance work with families, and that at times, FC had limitations with regard to serving families. This corroborates the quantitative results that case planner perceptions of FC limitations were associated with lower fidelity scores. The divergence of quantitative and qualitative results regarding organizational factors appeared to be related to supervisor knowledge and proactiveness. In the quantitative GLMM results, lower ratings of supervisor knowledge were related to higher fidelity scores. In contrast, interviews with case planners and the supervisor focus group indicated that supervisor knowledge was an important part of case planners’ ability to implement FC with fidelity. Additionally, across all three qualitative groups, the support of supervisors appeared to be an important factor in FC fidelity; however, this subscale was not significant in the GLMM analysis of organizational factors. Lastly, there was limited information to compare the level of education of the case planner and its effect on fidelity. In the leadership focus group, one agency leader stated they had found it important to hire master's level social workers to support FC replication; this in some ways mirrors the finding of lower fidelity among bachelor's trained case planners.

Discussion

Supervision matters

Results of the survey with case planners suggest that how proactive, knowledgeable, and perseverant a supervisor impact the fidelity of FC implementation. Proactiveness involves the supervisor developing an implementation plan, removing obstacles to implementation, and establishing clear standards for implementation. Perseverance includes reacting to critical issues and carrying on through implementation challenges and barriers (Aarons, Green, et al., 2014). Qualitative results suggest that in addition to supervisor proactiveness and perseverance, the availability of the supervisors, the opportunity to provide ongoing skill reinforcement, and knowledge of FC components were supportive of case planners’ ability to implement FC with fidelity. Some case planners described how their supervisor used coaching, role-plays, and modeling to reinforce skills, which is congruent with prior research (Bearman et al., 2013). Qualitative studies also corroborate the qualitative findings in this study that the content of supervision (Bertram et al., 2014), and general supervisory support and responsiveness were identified as contributors to fidelity (Kaufmann et al., 2012; Stern et al., 2008). Although little empirical research has connected supervisor practice to worker fidelity (Aarons et al., 2014), supervisor practice has been associated with child welfare workers’ job satisfaction (Barth et al., 2008) and intention to stay in the field (Johnco et al., 2014; Strolin et al., 2007), which could have an effect on a worker's ability to learn and improve intervention skills. Interviews with case planners and the focus group with supervisors corroborated that specific skills related to FC may require ongoing support throughout a case planner's tenure. However, in the mixed-methods analysis, there was a divergence between methods on supervisor knowledge. In the quantitative results, lower ratings of supervisor knowledge were related to higher fidelity scores. In contrast, interviews with case planners and the supervisor focus group indicated that supervisor knowledge was an important part of workers’ ability to implement FC with fidelity. Across all three qualitative groups, the importance of supervisors and their supportiveness appeared to be an important factor in supporting FC fidelity; however, this subscale was not significant in the quantitative analysis. Lastly, there was limited qualitative information on case planners’ level of education and its effect on fidelity. In the leadership focus group, one agency leader stated they had found it important to hire master's level social workers. This was supported by the quantitative analysis that those with Master's degrees had higher fidelity scores on average.

Quantitative results did not identify any other organizational results as significant contributors to workers’ fidelity. This is somewhat congruent with mixed results of organizational culture and climate's effect on fidelity (Beidas et al., 2014; Beidas et al., 2015); however, other studies have shown positive associations between quality improvement strategies (Crea & Crampton, 2011; Holth et al., 2011) and training (Whitaker et al., 2012) with fidelity. Organizational factors identified in the qualitative portion of the study and consistent with prior research include peer support (Chung et al., 2014), quality improvement strategies (Kaufmann et al., 2012; Stern et al., 2008), and training manuals (Chung et al., 2014; Stern et al., 2008) in supporting workers’ fidelity. Findings from the qualitative portion of the study suggest there may have been organizational factors important to fidelity not captured in the case planner survey related to aspects of leadership's direct support of practice, resource availability to assist with family needs, and external system policies such as time frames and case progress expectations.

In terms of individual characteristics, the quantitative portion of this study revealed that the level of education was associated with case planner fidelity. This result is somewhat surprising, given that degree was not a predictor of fidelity in previous research (Bearman et al., 2013;Whitaker et al., 2012). The focus group with leaders reiterated the value of Master's trained workers in achieving fidelity. However, in the focus group with supervisors, some suggested that the longer the worker was in child welfare, the more difficult it was for them to learn new FC skills. This is aligned with previous research that suggests that age was inversely related to worker fidelity (Whitaker et al., 2012), as well as one study that found that the more experience a worker had in months, the less likely they were to implement an intervention with fidelity (Beidas et al., 2015).

Case planner self-efficacy was not a significant predictor of fidelity which was surprising given the theoretical basis for exploring it (Durlak & DuPre, 2008). Perhaps this is because organizational and perceptions of intervention factors outplay how well a case planner believes they can implement the intervention. Mixed-methods analyses suggested that case planners with below average fidelity scores had mixed views on their self-efficacy in delivering the model, perhaps influencing the degree to which case planners were able to execute elements of the model. This could have also been influenced by a perceived fit of the model with families’ needs and situations.

Other individual factors related to fidelity of FC identified in the qualitative component were relational and technical attributes. Although they have not been directly connected with fidelity previously, some of these attributes have been identified as important for implementation such as attitudes toward monitoring, feedback, job security, and organizational support (Aarons, Cafri, et al., 2012). Less known is the degree to which relational attributes such as empathy, persistence, and passion related to fidelity. Qualitative data suggest that these relational attributes may influence a workers’ ability to persevere through implementation and also connect with the family and colleagues in a way that promotes skill attainment and fidelity. Understanding the role of relational attributes and skills in supporting or enhancing the fidelity of an intervention is an area for future research.

Quantitatively, perceptions of the intervention were captured through individual attitudes toward the fit, limitations, balance, burden, and job security of the EBP. Attitudes of limitations of the EBP were negatively associated with FC fidelity and the only significant individual factor overall. However, many of the case planner interviews suggested factors related to the intervention that went beyond an individual's perceptions of the intervention and focused more on the fit of the intervention within the system and particular characteristics of families served. Additionally, focus groups with supervisors implied that particular components of FC may require ongoing support through supervision to achieve fidelity. More complex components of intervention may require implementation support indefinitely in order to achieve fidelity.

It is important to note that there was a large amount of variability in the fidelity scores of case planners in the study. Although bootstrapping was used as a statistical strategy to control for the variability, the heteroskedastic characteristics of the data suggest that fidelity may be dynamic and a result of a variety of factors beyond this study. Fidelity scores in the study included results of three independent fidelity reviews, over a year and a half period. In an in-depth review of each individual fidelity scores, there were no obvious patterns in the data such as similar scores across workers, time periods, or agencies.

As with all research, this study has its limitations. There were likely several unmeasured factors related to worker performance, such as the appropriateness of cases referred for FC, the competing burdens associated with high caseloads, and the variability of families’ needs on a caseload. As acknowledged, bootstrapping was used to respond to the data heteroskedasticity; however, this is not the only possible option for dealing with the variability challenges. The sample sizes of the organizations were small, and future research should further investigate possible clustering effects, such as some sites achieving high fidelity more quickly than others. There may be other organizational variables such as degree of turnover, severity of case mix, and so on that could influence heteroskedasticity at the organizational level. There was substantial turnover in case planners between the case planners who had fidelity scores from previous fidelity reviews, case planners who completed the survey, and case planners who were available for follow-up interviews. This limited the sample and could have potentially introduced respondent bias.

Findings from this study reiterate the importance of supervision in child maltreatment prevention but extend practice implications by connecting perceptions of supervision practices directly to fidelity scores. Quantitative results suggest that the more proactive and perseverant a supervisor is in supporting a case planner, the more likely the case planner is to implement FC with fidelity. Qualitative results corroborate these findings but also suggest that the availability of a supervisor and the ongoing reinforcement of advanced skills may require ongoing supervisory support. The literature supports the assertion that ongoing supervision in social work is important (Hair, 2013; Mor Barak et al., 2009; Munson, 2002). Supervision may be a way to ensure frontline staff are able to implement consistent practice with some of the most vulnerable populations (Collins-Camargo & Groeber, 2003). In practice, ensuring that supervisors have the time and resources to meet with staff to provide substantial supervision is recommended based on the study findings. In addition, the focus group with supervisors suggested that just like case planners, supervisors also need ongoing skill development and reinforcement.

Although not identified in the quantitative portion of this study, qualitative results suggest that peer support, training, and leadership are additional supports case planners identified as influencing their ability to implement FC with fidelity. Supervisor focus groups also revealed coaching for supervisors as an important support. This multilevel support (case planner and supervisor) is an important distinction.

Findings from this study also suggest there are some systems-level policy and practices that may affect fidelity, such as how families are referred to FC and paperwork requirements by the child welfare system. These factors were not explored quantitatively in the study, and more research is needed to understand how these variables relate to fidelity. This is an important piece of the puzzle in understanding how fidelity of interventions occurs in real-world settings and the marrying of intervention and systems requirements (Proctor, 2012).

This study sought to understand the individual and organizational factors that affect the fidelity of interventions in community-based settings. A mixed-methods approach was used to unpack the complexities of implementation in typical prevention settings. The results indicate that real-world implementation is challenging and that to implement with fidelity, the supervisor is the primary support for workers and can promote their ability to persevere through challenges and apply intervention skills consistently across a range of families with varying needs. Ultimately, if the aim is for children and families to benefit from the investments made to develop and test interventions, then it is necessary to create aligned systems of support for workers so that each level of the system supports the effective practice.