Implementation fidelity in adolescent family-based prevention programs: relationship to family engagement

Abstract

Reliability and validity of intervention studies are impossible without adequate program fidelity, as it ensures that the intervention was implemented as designed and allows for accurate conclusions about effectiveness (Bellg AJ, Borrelli B, Resnick B et al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH behavior change consortium. Health Psychol 2004; 23: 443–51). This study examines the relation between program fidelity with family engagement (i.e. satisfaction and participation) in family-based prevention programs for adolescent alcohol, tobacco or other drug use. Families (n = 381) were those with an 11- to 12-year-old child enrolled in Kaiser Permanente in the San Francisco area. Families participated in one of two programs: Strengthening Families Program: For Parents and Youth 10-14 (SFP) (Spoth R, Redmond C, Lepper H. Alcohol initiation outcomes of universal family-focused preventive interventions: one- and two-year follow-ups of a controlled study. J Stud Alcohol Suppl 1999; 13: 103–11) or Family Matters (FM) (Bauman KE, Ennett ST. On the importance of peer influence for adolescent drug use: commonly neglected considerations. Addiction 1996; 91: 185–98). Fidelity was assessed by: (i) adherence to the program manual and (ii) quality of implementation. No relationships were found for FM, a self-directed program. For SFP, higher quality scores were related to higher parent satisfaction. Higher adherence scores were related to higher satisfaction for youth, yet surprisingly to lower satisfaction for parents. Parent sessions involve much discussion, and to obtain high adherence scores, health educators were often required to limit this to implement all program activities. Findings highlight a delivery challenge in covering all activities while allowing parents to engage in mutually supportive behavior.

Introduction

Implementation fidelity of prevention/intervention programs refers to the degree to which program facilitators deliver the intervention as intended by program developers [1–3]. Methods of fidelity assessment include measuring different aspects of the program delivery, including adherence to program guidelines or manuals and the quality with which the program was delivered. Specifically, adherence refers to whether the key components of the program are being delivered as they were designed, whereas the quality of program delivery includes the skill level of the providers, as well as provider characteristics such as their enthusiasm and preparedness in delivering the intervention [4].

Components of evidence-based prevention programs are usually designed by program developers based on extensive testing and theoretical models linked to outcomes [2]. As a result, modifying or deleting components could reduce program impacts on outcomes [2]. In addition, ensuring the reliability and validity of intervention studies is impossible without adequate program fidelity, as it ensures that the intervention was implemented as designed and allows for accurate conclusions about effectiveness [5]. However, controversy exists as to whether some minor adaptations to programs are natural and may even be desirable [6, 7]. In contrast, there is the view that changes risk reducing program effects [8–10].

Prior research strongly suggests that some programs or interventions are only effective when they are implemented with high levels of fidelity [11–13], thus making it a critical component of intervention studies, particularly when testing for efficacy of the intervention. Similarly, Schoenwald's study [14] of adherence in multi-systemic therapy (MST) for youth with drug abuse and behavioral problems found that greater adherence fidelity was related to better short- and long-term outcomes. Likewise, Henggeler et al. [15] found that outcomes for violent and chronic juvenile offenders and their families were considerably more favorable when adherence fidelity to MST was high. However, other studies of substance abuse treatment have reported curvilinear as well as linear relationships between fidelity and outcomes in that moderate levels of fidelity were related to the most improvement in outcomes, with low and high levels of fidelity linked to smaller improvement [16, 17]. The authors of these studies suggest that moderate levels of fidelity may lead to the greatest improvements because low fidelity indicates that the core components of the intervention are not being implemented, but high fidelity may indicate that not enough flexibility exists to respond to client needs [16, 17].

Although prevention studies often report program fidelity, research on this issue in the prevention area is limited [7, 13]. However, there are a few studies that demonstrate the importance of implementation fidelity for prevention programs and link these findings to program outcomes [18–23]. Fidelity is associated with changes in mediators of program outcomes in that better delivery quality has been related to better scores on program mediators [24, 25]. Most of these studies, however, examine only school-based prevention programs.

The active engagement of participants is also critical to prevention program success [26]. However, successfully engaging targeted participants often presents a significant challenge, especially for family-based programs, and points to the need for studies of influences on participant engagement [26–28]. Satisfaction with the program and active participation could be considered indicators of engagement. It is important to determine what factors might impact the ability to achieve high participant satisfaction and encourage families to complete the program. Few studies investigate the importance of fidelity for encouraging continuing program participation and participant satisfaction with the program, and family-based prevention studies are particularly lacking. Yet, one of the reoccurring issues for family-based strategies is the engagement of families [26, 27]. If a program is delivered with greater levels of warmth and program facilitators are highly skilled, greater satisfaction with the program and greater likelihood in continued participation are expected. However, the importance of program fidelity for program participation and satisfaction in prevention programs is largely unstudied. For example, McHugo et al. [29] implemented an assertive community treatment program for adults and found no difference in the satisfaction of patients enrolled in high-fidelity versus low-fidelity programs, yet better program retention for those in high-fidelity programs. Yet, due to the small sample size (n = 87) and adult population, it is difficult to extend findings to adolescent prevention programs. Further, no known studies have examined these issues in family-based prevention programs.

The current study examined the relationships of program fidelity with family satisfaction and participation in two family-based prevention programs for adolescent alcohol, tobacco or other drug (ATOD) use. We hypothesized that higher levels of program fidelity would be related to improved rates of both family satisfaction and participation. We expected that programs implemented with higher adherence to program manuals and with greater skill would result in more satisfied families and continued participation.

Methods

Sample and procedures

This study is part of a longitudinal study that examines the effect of choice on recruitment, participation and adolescent outcomes for two family-based ATOD prevention programs implemented in a health care system. Families were identified through one of four Kaiser Permanente (KP) medical centers in the San Francisco Bay Area. Before contact, families were randomized into one of two conditions: (i) randomized control trial (RCT) in which families were assigned to one of two prevention programs or a control group or (ii) choice condition, in which families could choose between the programs. A total of 614 families were recruited and completed baseline enrollment interviews after completing informed consent procedures. Separate, private face-to-face interviews were conducted with mother and child, who each received $30 for interview completion. Of the enrolled families, 496 families enrolled in intervention groups (i.e. excluding the control group). This study focuses on the 381 intervention group families who participated in at least one session or completed at least one booklet of the prevention program they enrolled in, as non-participating families would not be exposed to program fidelity. Study procedures were approved by the Institutional Review Boards at the Pacific Institute for Research and Evaluation and Kaiser Foundation Research Institute.

Inclusion/exclusion criteria

To be eligible for the study, at least one family member must have been a member of KP at the time the sample was drawn, and the family must have included an 11- to 12-year-old child. Family members must also be able to speak English as the programs were only offered in English during the study. As both programs are universal prevention programs, neither targeted toward families already experiencing behavioral problems nor intended for treatment of problems, families whose child was currently in treatment for ATOD were excluded. Only mothers or female caregivers and the child were enrolled in the study and responded to study surveys. However, fathers and other family members were also encouraged to participate in the program activities.

Sample

About half (49.3%) of the youths were female. Youths were 11 or 12 years of age at recruitment (M = 11.5; SD = 0.51). Youths were allowed to endorse multiple ethnicities, resulting in the following breakdown: 62.5% White, 22.8% Asian, 18.6% Hispanic/Latino, 16.1% African-American, 8.4% Native American, 2.9% Pacific Islander and 12.9% unknown. Over half (60.9%) of mothers had graduated from college. Over three-fourths (78.4%) of parents were married and reflected diverse income levels: 8.4% had family income of $40 000 or less, 29.6% had income between $40 001 and $80 000, 28.6% were between $80 001 and $125 000, 12.1% were between $125 001 and $150 000, 11.3% were between $150 001 and $200 000, 7.0% were between $200 001 and $300 000 and 3.0% had incomes above $300 000.

Programs

The two programs offered were Strengthening Families Program: For Parents and Youth 10-14 (SFP) [30] and Family Matters (FM) [31]. Both programs are theory-based prevention programs that have previously demonstrated effective outcomes [32, 33]. Both are universal programs and address similar risk factors for adolescent ATOD use. These two programs are among a small number of family-based programs that have been rigorously evaluated in general populations with randomized experimental designs and found to reduce adolescent ATOD use. The two programs differ substantially in structure, demand on families and program cost to allow a meaningful choice to parents. These important differences also allow for the identification of different challenges in implementation fidelity for programs with differing formats and family demand, including the identification of different relationships between fidelity and engagement.

SFP.

SFP is based on the biopsychosocial model and stresses the importance of other family risk and protective factors that have empirical support for adolescent ATOD use and other problem behaviors [34–36]. SFP targets risk factors such as poor parental discipline skills, poor quality parent–adolescent relations and protective factors such as parental empathy and parent–adolescent bonding [37, 38]. SFP is a structured interactive group program facilitated by health educators during seven weekly sessions at KP medical center sites. The first hour of every session consists of separate groups for parents and youths. Parent sessions include interactive discussions and observation of videotaped actors modeling problems and strategies. Parent sessions focus on areas such as appropriate behavior management skills considering the youth's developmental stage, family-problem-solving skills, parental influences, effective communication and communication of expectations for the youth's ATOD use. Youth sessions include games and activities and focus on pro-social goal setting, coping skills, understanding parental stressors and responsibilities, importance of family cooperation and peer pressure resistance strategies. The second hour consists of a family session in which families have the opportunity to practice skills learned during the first hour.

FM.

FM is based on fundamental concepts of public health and health promotion practice such as the promotion of healthy lifestyles and risk/protection through ecological or environmental change [39, 40] and was designed to address alcohol and tobacco use among adolescents [41]. In contrast to SFP, FM is a parent-directed home-based program involving four booklets mailed one at a time to parents, in which parents read the materials and then lead the activities with their child. Health educators make phone contact with parents to address any issues they have with the program and to encourage completion of the activities. Health educators only speak with the parent so that the parent is leading the program for their family. As only mothers/female caregivers were enrolled in our study, health educators only had contact with her, and she then led the program for her child and any other adults in the family who also participated. Booklets include topics such as negative consequences of ATOD use, adolescent vulnerability to such use and family and outside influences on adolescent use of alcohol and tobacco. Activities include some activities for adult family members only such as listing things they do that might inadvertently encourage the adolescent's ATOD use. Examples of family activities include meeting to agree upon rules and sanctions for adolescent use, practicing what to do if a friend offers alcohol or tobacco and watching favorite television shows to discuss alcohol- and tobacco-related messages. The parent leads the program activities with the adolescent and other family members during times that fit their own schedule.

As described elsewhere [A. E. Aalborg, B. A. Miller, G. Husson et al. Unpublished material], there was extensive initial and ongoing training of health educators for the delivery of the two programs. Initial program training was conducted by the original developers of SFP and FM through modeling of program delivery, role-playing and training manuals. It is important to note that both of these programs were designed for the prevention of adolescent ATOD use and not for treatment of existing problems. Recruiters and health educators emphasized this to parents and explained that the programs would help build strengths in families to reduce or avoid later problems. However, if participants did approach staff with problems, whether related to ATOD use or other issues, they were given a list of resources that could provide help and were connected with the resources of KP.

To provide an objective record of program delivery for both programs, all SFP sessions were videotaped and all FM telephone calls were automatically audio recorded with the Telestat system. A series of rating systems adapted from the program originators' manual and original fidelity forms were used to determine fidelity [30, 42]. Separate staff members were trained as fidelity raters so as to provide objective assessments of program implementation. Fidelity raters were trained in both the program delivery and the use of the rating forms.

Training involved several steps. First, raters used a fidelity rating manual to learn rating guidelines and then observed or listened to recordings (video recording for SFP and audio recording for FM) together with the principal investigator and other senior staff and discussed guidelines for how items should be rated. The project director rated a group of sessions that were used for training. Once raters understood rating protocol, they were then assigned to rate sessions independently which the project director had already rated. Once raters could achieve 85% consensus with the project director, they were allowed to rate sessions independently. Interrater reliability was measured by calculating kappa and a consensus score, which was the percentage of items agreed upon by the raters. The average consensus scores were 92% for SFP (kappa = 0.67) and 90% for FM (kappa = 0.76). Approximately one-third of the tapes were double-rated to ensure consistency in rating and assess interrater reliability. Raters also kept notes to provide continuing feedback to health educators for ongoing training.

Measures

Fidelity scores

Two components of fidelity were assessed as raters reviewed a sample of videotapes (SFP) or audiotapes (FM): (i) adherence and (ii) quality of implementation. Adherence addressed how closely the health educators delivered the program according to the program manual. Quality of implementation addressed how well the health educator implemented the program with characteristics such as clarity, warmth and building rapport with participants. Separate fidelity instruments were used for SFP and FM, due to the substantially different nature of the program structures.

Adherence.

For SFP, fidelity observation rating instruments developed by the originators of the SFP program were adapted for the current study. Adherence items assessed adherence to the manual, such as whether specific activities in each session were conducted exactly as described in the manual and include following the time frame for each activity. For example, one activity, allotted 4 min, involved four steps: (i) have parents list small privileges to take away, (ii) tell parents that using chores and taking away privileges will not necessarily work right away, (iii) tell parents that the best privileges to remove are short and easy to keep track of and (iv) ask parents what could happen if they give a large penalty for small misbehavior. Items rated varied across sessions depending on the specific activities in each session. Each session included items rating the adherence to the manual by following steps in the activities (0 = no and 1 = yes for each step). A score of 100% would indicate that health educators implemented every single step of every activity. Although the detailed program manual gave clear recommended steps for the program deliverer, the failure to follow each of the specific steps is not equated with failure to deliver the content of the program. Rather, it may reflect the lack of time to complete every single step with a group and/or a slight variation in the recommended steps. Some items also included a count of the number of points the health educator raised on a specific topic. Items such as these were coded so that 0 = none and 1 = any, to reflect adequate implementation. The number of items rated varied for each session based on the number of activities and steps within each activity in that session. A summary score for adherence was created for each session by summing the items and dividing by the total possible score for each session.

For FM, observers evaluated adherence fidelity using rating forms based on the FM manuals. Raters scored a random sample of approximately 25% of telephone interviews (at least one book per family), calculating a fidelity summary rating of 15 items. Items included five questions assessing adherence to the manual (0 = no and 1 = yes) such as checking for receipt of program materials. Adherence fidelity scores were calculated for each family by summing item scores and dividing by the total possible scores.

Quality of implementation.

For SFP, 16 items assessed the quality of implementation. Six of these evaluated the effectiveness of health educator delivery such as clarity in explanations, friendliness and controlling the pace. Ratings were 0 = inadequate and 1 = adequate. Six of the items assessed inappropriate group leader processes such as acting uninterested or being critical of participants' ideas (0 = frequently and 1 = never). Three of the items assessed group participation such as level of responsiveness of participants. Rating scales were also 0 = inadequate and 1 = adequate. One item assessed whether the room setup was conducive to proper delivery (0 = no and 1 = yes). A summary quality score was created for each session by summing the items and dividing by the total possible score. Scores for each session were averaged to create the quality score for each type of session (i.e. family, parent and youth). Scores from each session type were then averaged to create a quality score overall across session types. This overall score was used in analysis.

For FM, 10 items assessed the quality of implementation, such as level of rapport with parent and enthusiasm. The scale was 0 = inadequate and 1 = adequate. Quality fidelity scores were calculated for each family by summing item scores and dividing by the total possible scores.

Family engagement

Engagement was measured by (i) family satisfaction and (ii) participation.

Family satisfaction.

For SFP, parent and adolescent satisfaction data were collected through the use of anonymous surveys administered at the end of the final session. Parents were asked to assess how much they agreed or disagreed with 12 questions assessing program impact and delivery quality. Youths were asked to assess nine questions. Because ratings were anonymous and could not be linked to individuals, mean scores for each item were created for each SFP group, and overall parent and youth satisfaction scores were then created for each group by averaging all items (Cronbach's α = 0.91 and 0.83, for parent and youth, respectively).

Satisfaction data for FM were collected by health educators in their review of each booklet with the parent. Parents were asked items reflecting: (i) how much they liked the activities (one item per activity on a four-point scale where 1 = not very much and 4 = a lot), (ii) after completion of each book, how important they felt it was to do FM (0 = less important, 1 = neither and 2 = more important) and (iii) how confident they were that they could prevent their child from using alcohol or tobacco (0 = less confident, 1 = about the same and 2 = more confident). In addition, health educators rated the parents' enthusiasm for the program at the end of each booklet, on a four-point scale (1 = not at all enthusiastic and 5 = very enthusiastic). A summary score was created for overall satisfaction with the program (Cronbach's α = 0.73) by standardizing (due to differing item response scales) and then averaging items.

Participation.

Program participation was assessed with a percentage (0–100%) indicating the amount of the program completed. This percentage was based on the number of FM booklets completed (0–4) or the number of SFP sessions completed (0–7).

Background variables

Background variables include youths' reports of ethnicity, mothers' reports of their education (1 = college or more and 0 = high school or less), marital status and family income and youths' reports of their gender and age.

Results

Descriptive analyses

Fidelity of programs

Health educators for both programs delivered the programs at the level of quality established by the developers of the programs.

Adherence.

For SFP, on average, health educators followed 78% of the steps as outlined in the SFP program manual. For FM, on average, 82% of the program was adhered to.

Quality of implementation.

For SFP, 93% of sessions rated had quality scores that met or exceeded expectations. For FM, 91% of calls met or exceeded criteria for quality scores.

Family engagement

Family satisfaction.

Descriptive statistics for family satisfaction are shown in Table I. For SFP, the mean session score for parent satisfaction was 3.32 (SD = 0.19) and youth satisfaction was 3.18 (SD = 0.24) out of a scale ranging from 1 to 4, indicating that parents and youths were very satisfied with the program on average. For FM, average rates of satisfaction were 3.43 (SD = 0.37) on a 1–4 scale for how much the parent liked the activities, 1.85 (SD = 0.31) on a 0–2 scale for importance of the program, 1.75 (SD = 0.34) on a 0–2 scale for confidence in preventing their child's use of alcohol or tobacco and 4.04 (SD = 0.73) on a 1–5 scale for health educator ratings of parent enthusiasm for the program. Ratings indicate that parents were very satisfied with the program overall.

Table I.

Descriptive statistics for family satisfaction

	Minimum	Maximum	M	SD
Parent SFP satisfaction	2.95	3.55	3.32	0.19
Youth SFP satisfaction	2.70	3.54	3.18	0.24
Liked FM activities	2.33	4.00	3.43	0.37
Perceived importance of FM	0.00	2.00	1.85	0.31
Confidence in FM to prevent youth's ATOD use	0.25	2.00	1.75	0.34
FM parent enthusiasm	1.75	5.00	4.04	0.73

Participation.

This study includes only those families who completed at least one booklet of FM or attended one SFP session. Of these families, those in the FM programs had an average completion rate of 83.94% of the program (close to all four booklets). About three-fourths (74.1%) of families completed all four booklets, whereas 15.1% completed only one booklet. Families in the SFP programs attended 71.05% of the total sessions (about five of the seven sessions) on average. More than half (53.7%) attended six or seven of the sessions, whereas 8.1% attended only one session.

Condition differences

Since families in the larger study had been assigned to one of two conditions, (i) choice, in which they could choose between the two programs, or (ii) RCT, in which they were assigned to one of the two programs or to a control group, we also examined differences according to condition. For the FM program, no differences existed between conditions for fidelity, satisfaction or participation levels. Specifically, 83% of the program was adhered to for the choice condition and 82% for the RCT condition (t = 0.91, df = 176, P = 0.36). For both conditions, quality fidelity scores showed that 91% of the calls met or exceeded criteria (t = −0.23, df = 176, P = 0.82). Neither the satisfaction scores (t = −0.19, df = 225, P = 0.85) nor the participation levels (t = −0.33, df = 230, P = 0.74) significantly differed, with choice families completing 83.5% of the program and RCT families completing 84.7%.

For SFP, no differences existed for adherence fidelity scores between the two conditions, with 77% adhered to for choice families and 79% for RCT families (t = −1.46, df = 110.25, P = 0.15). However, quality scores were significantly different between conditions, with slightly higher scores for RCT groups (94% of sessions meeting or exceeding criteria versus 92% for choice families; t = −2.37, df = 135.33, P < 0.05). SFP satisfaction scores did not significantly differ between conditions for youths (t = −0.43, df = 93.83, P = 0.67), but did for parents (t = 5.56, df = 121.90, P < 0.001), with parents in the choice condition being more satisfied than RCT parents (3.41 versus 3.25). There were no significant differences in participation rates for the two conditions (t = 1.18, df = 147, P = 0.24), with choice families attending 73.6% of sessions and RCT families attending 68.2%. Due to differences between conditions for the SFP program, we controlled for condition in our regression analyses.

Regression analyses

Family engagement

Family satisfaction.

Multiple regression analyses were conducted, controlling for background demographic variables. For the SFP program, condition (1 = choice and 2 = RCT) was also controlled for. As presented in Table II, there were differences across parent and youth satisfaction as related to program fidelity results. For SFP, higher overall quality scores were related to greater parent satisfaction (β = 0.61, P < 0.001). Overall quality scores did not impact youth satisfaction (β = −0.09, P = 0.57). In contrast, higher adherence scores were positively related to higher satisfaction for youth (β = 0.48, P < 0.01), but negatively related to parent satisfaction scores (β = −0.38, P < 0.01).

Table II.

Multiple regressions for the relationship between overall SFP fidelity scores with parent and youth satisfaction

	Parent SFP satisfaction				Youth SFP satisfaction
	B	SE	β	t	B	SE	β	t
Marital status	−0.03	0.04	−0.06	−0.77	0.01	0.05	0.01	0.09
Child age	−0.04	0.03	−0.10	−1.26	0.00	0.04	−0.01	−0.09
Child gender	−0.01	0.03	−0.04	−0.47	−0.02	0.05	−0.05	−0.51
Family income	0.01	0.01	0.05	0.58	0.00	0.02	0.02	0.17
Education	−0.01	0.01	−0.10	−1.18	0.01	0.02	0.03	0.32
White	0.02	0.03	0.06	0.71	0.07	0.05	0.13	1.42
Site	0.00	0.02	0.00	0.00	−0.09	0.03	−0.37	−2.67**
Condition	−0.17	0.03	−0.45	−5.61***	−0.01	0.05	−0.01	−0.13
Overall SFP adherence fidelity score	−0.86	0.32	−0.38	−2.68**	1.40	0.49	0.48	2.88**
Overall SFP quality fidelity score	3.19	0.65	0.61	4.89***	−0.57	0.99	−0.09	−0.57

**P < 0.01, ***P < 0.001.

No relationship between FM fidelity scores and program satisfaction was detected. Specifically, neither FM adherence scores (β = 0.09, P = 0.25) nor FM quality scores (β = −0.06, P = 0.46) were associated with overall satisfaction with the program.

Participation.

Fidelity scores were unrelated to program participation for both programs. Specifically, for SFP, neither quality (β = 0.13, P = 0.35) nor adherence scores (β = −0.28, P = 0.09) were related to participation. Likewise, for FM, neither quality (β = 0.12, P = 0.13) nor adherence (β = −0.14, P = 0.08) scores were related to participation.

Discussion

Findings from this study show that fidelity influenced the two prevention programs in different ways. In fact, there was no evidence that fidelity was related to engagement for the self-directed program, FM. In contrast, findings suggest a complex relationship for the group program, SFP, in which differences were found in the response to the two aspects of fidelity assessed, adherence and quality.

For the SFP program, higher quality scores were related to higher parent satisfaction, whereas higher adherence scores were related to higher satisfaction for youth, yet lower satisfaction for parents. Different parent and youth findings may have emerged due to the differing formats of the parent and youth sessions. Parent sessions involve much discussion and tightly timed activities that health educators must cover. Most parent sessions involve the use of videos that guide the activities, and the video stops at certain points where the health educator is to lead the parents in discussion of the prompt shown on the screen. In these cases, the video displays a countdown timer and then resumes once the allotted time ends. To obtain high adherence scores, health educators were often required to limit discussions to implement all program activities. In addition to allowing inadequate time to complete remaining activities, stopping the video (to allow longer discussions) also directly lowered adherence scores. Youth sessions differ in that there is less discussion, with more time allotted per activity, thus making control of the pace less of an issue for health educators.

A delivery challenge exists for the parent sessions in finding ways to cover all program activities while allowing participants to engage in mutually supportive behavior. A prior review of research on implementation fidelity in prevention programs found that not implementing all the program's core components is a common adaptation made by local deliverers [1] and is commonly done due to time demands [2]. Another study of fidelity in the SFP program found that in addition to reasons including time, some adaptations were made in order to facilitate group dynamics and build rapport with the audience [2].

This finding highlights the debate over the importance of local adaptation versus strictly adhering to the program manual [43]. Although some propose that local deliverers negotiate modifications to balance these issues [6], others find problems with this approach [43]. For example, Elliott and Mihalic [43] contend that it is difficult for local deliverers to determine which program components can be modified without altering effectiveness and that allowing modifications is not necessary to get investment by deliverers. Further, evidence strongly supports that many programs work better when adapted with greater levels of fidelity [11–13]. Perhaps the focus should be on training health educators to deal with issues of time demands. Health educators could be trained to more skillfully redirect participants so that they can keep the pace moving to cover all content, yet still make participants feel they are able to gain support from one another through discussion. Alternatively, these findings also raise the possibility that parent SFP sessions might be more positively received if the structure of the parent sessions was changed so that more time was allowed for fuller discussions of issues in each activity.

For the FM program, neither type of fidelity scores were related to parental satisfaction with the program. In FM, only parents reported their satisfaction with the program, due to the differing structure and format of the program (only parents communicate directly with health educators). The group setting for SFP may have led parents to expect more time to talk and interact with other parents, accounting for their differential levels of satisfaction when adherence fidelity was higher. In addition, FM is a home-based parent-directed program, making the role of health educator less central to delivery of the program and less likely for fidelity by health educators to have an influence on parent satisfaction.

Neither SFP nor FM fidelity scores were related to program participation. Other factors may have played a more important role in whether the family participated or not, such as scheduling conflicts among family members and time limitations. Time constraints have been shown to be a major influence on participation in preventive interventions [44], and in fact, this was the most common reason mentioned for not completing booklets or attending sessions as reported to health educators. Other work from this study is exploring issues and barriers to participation and satisfaction in prevention program implementation [A. E. Aalborg, B. A. Miller, G. Husson et al. Implementation of two adolescent family-based ATOD prevention programs in health care settings: comparisons across conditions and programs. Unpublished material, 2009; A. E. Aalborg, B. A. Miller, G. Husson et al. Parent, adolescent and program characteristics related to participation in family-based adolescent alcohol prevention programs. Unpublished material, 2009].

Since the current study was part of a larger project examining the influence of choice, we were also able to examine differences in fidelity, satisfaction and participation between the choice and RCT (assigned) conditions. There were no differences between conditions in these variables for the FM program. For the SFP program, adherence fidelity was not significantly different between the two conditions, but quality fidelity was significantly better for families in the assigned condition. Although significant, this does not appear to be a meaningful difference, as quality fidelity was very high in both conditions (94% RCT versus 92% choice). SFP participation was not different between conditions. Youth satisfaction did not differ between conditions, but parents were more satisfied in the choice condition. This finding is consistent with prior research showing that participants feel more invested and more content with their treatment when they have involvement in treatment decisions [45].

Limitations of the current study exist. Results are from two specific family-based prevention programs, so it is unknown whether findings would generalize to other family-based prevention programs. However, the two programs in this study reflect two very different program formats, with one being directed by parents at home and the other involving regularly scheduled group sessions. As such, findings can inform other programs as well.

Overall, results suggest that two aspects of fidelity, adherence and quality, have important relationships with how families experience interactive group prevention programs, especially in regards to their satisfaction with the program. Findings suggest that health educators should be encouraged to deliver prevention programs with a high degree of warmth and clarity as well as adherence to program design. However, careful training is needed to help health educators develop the ability to implement all program components while promoting mutual participant support through active discussions.

Finding determinants of program satisfaction could be an aid in program dissemination because if participants have greater levels of satisfaction, they may be more likely to recommend the program to family and friends [46] or to request the program at their medical facility. Work is also currently underway to examine whether fidelity, in addition to other constructs, is related to program-targeted outcomes. Because fidelity was related to satisfaction differently for parents and youths, future studies that examine specific factors driving these differences would be important for informing delivery of prevention programs. For example, qualitative studies examining the participants' perceptions of factors that influenced their satisfaction and participation would contribute to an understanding of these issues.

Funding

National Institute on Alcohol Abuse and Alcoholism (R01-AA015323-01, 2005-2010, ‘Adolescent Family-Based Alcohol Prevention’, to B.A.M.).

Conflict of interest statement

None declared.