Fidelity issues in secondhand smoking interventions for children

Abstract

This paper reviews methodological and theoretical fidelity of secondhand smoking (SHS) intervention studies (n=29) that target protection of children in their home. In 2005, interventions were evaluated in terms of treatment fidelity according to guidelines provided by Borrelli et al. of the National Institutes of Health Behavior Change Consortium. The degree of fidelity was evaluated based on the percentage of criteria met; the inter-rater reliability based on percent agreement across independent raters was 0.78. Analysis indicated that studies with higher treatment fidelity were more likely to obtain statistically significant results (p=.003) with the average fidelity rating of 0.74 for statistically significant studies vs. 0.50 for statistically non-significant studies. Higher treatment fidelity was also significantly associated with being a more recent investigation (year 2000 or later), an efficacy as compared to effectiveness trial, more intensive as compared to less intensive intervention, a trial in the U.S. as compared to foreign nations, and having a theoretical basis. After taking all other variables into account, only treatment fidelity was significantly related to study outcome (p=.052). Ratings of treatment fidelity were ranked and compared to previous rankings based on 342 behavioral change interventions; the rank-ordered correlation between previous and current ratings was 0.84, although median fidelity ratings were 0.10 points lower in the previous than in the present study (0.52 vs. 0.62; intraclass correlation=0.79). Improvements to the treatment fidelity evaluation guidelines were suggested, including the consideration of theoretical fidelity. Enhancing methodological and theoretical fidelity will speed identification of valid theoretical precepts that will, in turn, guide effective public health prevention programs.

Introduction

Secondhand smoke (SHS) is a known human carcinogen for which there is no safe level of exposure (U.S. Department of Health and Human Services [USDHHS], 2006; U.S. Environmental Protection Agency, 1993). Based on 1997 estimates of cigarette smoking prevalence worldwide, the World Health Organization (1999) estimated that about half of all children live in a home with one or more smokers. Healthy People 2010 (USDHHS, 2000) guidelines recommend that no more than 10% of children be exposed to SHS by the year 2010; however, in the U.S. approximately 23% of children are exposed to SHS in their homes (USDHHS, 2006). Children are particularly vulnerable to SHS exposure, with most exposure occurring in the home (Ashley & Ferrence, 1998). Exposure in cars is also common, and may be more harmful (Emerson et al., 1994). Children aged 3 to 19 years have higher levels of cotinine (a metabolite of nicotine and common biomarker of SHS exposure in nonsmokers) than individuals aged 20 years and above (USDHHS, 2005) and median levels of cotinine among children are estimated to be more than twice that of nonsmoking adults (USDHHS, 2006).

Since our most recent literature review (Gehrman & Hovell, 2003), 10 intervention studies targeting SHS exposure in children have been published (see Table 2). The studies were identified in January 2008 using MEDLINE and PsycINFO search engines, using the keywords environmental tobacco smoke, secondhand smoke, passive smoke/smoking and smoke pollution, and the same terms crossed with the search term intervention. Studies were selected if they were published in English, involved an intervention that targeted SHS exposure, and measured SHS exposure at baseline and at least once following the intervention. Interventions could target children’s SHS exposure directly by modifying parents and children’s exposure behavior or by instituting a home smoking ban, or indirectly by targeting parental smoking cessation. The studies were published between 1987 and 2006. The studies were evaluated in terms of treatment fidelity. Treatment fidelity is the degree to which intervention procedures adhere to the planned protocol and theoretical model (Borrelli et al., 2005; Moncher & Prinz, 1991; Resnick et al., 2005).

Table 2.

Treatment fidelity ratings of secondhand smoke reduction interventions targeting children.

	Woodward, 1987	Chilmonczyk, 1992	Vineis, 1993	Meltzer, 1993	Strecher, 1993	Greenberg, 1994	McIntosh, 1994	Hovell, 1994	Wall, 1995	Ericksen, 1996
Treatment design
1. Info about treatment
A. Length of contact	•	•	•	•	•	•	•	•	•	•
B. Number of contacts	•	•	•	•	•	•	•	•	•	•
C. Content of treatment	•	•	•	•	•	•	•	•	•	•
D. Duration of contact  over time	•	•	•	•	•	•	•	•	•	•
2. If more than one  treatment, equal  description	•	–	–	–	–	–	–	•	–	–
3. Info about control or comparison condition
A. Length of contact				–						•
B. Number of contacts	•	•		–	•	•	•	•	•	•
C. Content of treatment	•			–	•	•	•	•	•	•
D. Duration of contact  over time	•	•	•	–	•	•	•	•	•	•
4. Provider credentials	•			•					•
5. Theoretical model	•			•	•	•	•	•	•
Provider training
1. How trained		•							•
2. Standardized training		•			•				•
3. Provider skills assessed
4. Provider skill  maintenance					•	•			•
Delivery of treatment
1. Content delivered	•	•		•	•		•	•
2. Dose delivered		•			•	•	•		•
3. Provider adherence to  intervention					•	•			•
4. Non-specific treatment  effects								•
5. Treatment manual used				•				•
Receipt of treatment
1. Participant  comprehension  assessed			•	•	•	•		•
2. Method for improving  comprehension				•	•			•	•
3. Performance skills  assessed				•	•	•		•	•
4. Strategy to improve  performance				•	•			•
Enactment of treatment skills
1. Skills assessed in  relevant settings	•	•	•	•	•	•	•	•	•	•
2. Improve performance in  relevant settings				•	•	•	•	•	•
Percent of fidelity elements described	.46	.44	.28	.67	.76	.60	.48	.69	.68	.36
Study statistically significant				•	•	•	•	•	•
	Severson, 1997	Wahlgren, 1997	Irvine, 1999	Groner, 2000	Hovell, 2000	Emmons, 2001	Wilson, 2001	Wakefield, 2002	Hovell, 2002	Curry, 2003
Treatment design
1. Info about treatment
A. Length of contact	•	•		•	•	•		•	•	•
B. Number of contacts	•	•	•	•	•	•	•	•	•	•
C. Content of treatment	•	•	•	•	•	•	•	•	•	•
D. Duration of contact  over time	•	•	•	•	•	•	•	•	•	•
2. If more than one  treatment, equal  description	–	•	–	•	–	–	–	–	–	–
3. Info about control or comparison condition
A. Length of contact								•		•
B. Number of contacts	•	•	•	•	•	•	•	•	•	•
C. Content of treatment	•	•	•	•	•	•	•	•	•	•
D. Duration of contact  over time	•	•	•	•	•	•	•	•	•	•
4. Provider credentials	•			•	•	•			•	•
5. Theoretical model	•	•		•	•	•	•		•	•
Provider training
1. How trained	•				•	•			•	•
2. Standardized training	•				•	•			•	•
3. Provider skills assessed					•	•			•	•
4. Provider skill  maintenance	•				•	•			•	•
Delivery of treatment
1. Content delivered	•	•			•	•		•		•
2. Dose delivered	•				•	•		•	•	•
3. Provider adherence to  intervention	•				•	•			•	•
4. Non-specific treatment  effects		•			•				•	•
5. Treatment manual used		•		•			•			•
Receipt of treatment
1. Participant  comprehension  assessed		•			•	•	•	•	•	•
2. Method for improving  comprehension	•	•	•	•	•	•	•	•	•	•
3. Performance skills  assessed		•			•	•	•	•	•	•
4. Strategy to improve  performance	•	•		•	•	•	•	•	•	•
Enactment of treatment skills
1. Skills assessed in  relevant settings	•	•	•	•	•	•	•	•	•	•
2. Improve performance  in relevant settings	•	•	•	•	•	•	•	•	•	•
Percent of fidelity elements described	.76	.69	.36	.58	.92	.88	.56	.60	.92	1.0
Study statistically significant	•	•			•	•			•	•
	Conway, 2004	Fossum, 2004	Zakarian, 2004	Klinnert, 2005	Abdullah, 2005	Loke, 2005	Chan, 2005	Kallio, 2006	Yilmaz, 2006	Percent Fidelity
Treatment design
1. Info about treatment
A. Length of contact				•	•	•	•		•	0.79
B. Number of contacts	•	•	•	•	•	•	•	•	•	1.00
C. Content of treatment	•	•	•	•	•	•	•	•	•	1.00
D. Duration of contact over  time	•	•	•	•	•	•	•	•	•	1.00
2. If more than one  treatment, equal  description	–	–	–	–	–	–	–	–	•	1.00
3. Info about control or comparison condition
A. Length of contact					•	•	•		•	0.25
B. Number of contacts	•	•	•	•	•	•	•	•	•	0.96
C. Content of treatment	•	•	•	•	•	•	•	•	•	0.93
D. Duration of contact over  time	•	•	•	•	•	•	•	•	•	1.00
4. Provider credentials	•	•	•		•	•	•			0.52
5. Theoretical model	•	•	•	•	•	•				0.72
Provider training
1. How trained	•	•	•		•	•	•			0.45
2. Standardized training	•	•	•		•	•	•			0.48
3. Provider skills assessed	•	•	•		•					0.28
4. Provider skill  maintenance		•	•		•					0.38
Delivery of treatment
1. Content delivered		•	•	•	•	•	•			0.62
2. Dose delivered		•	•	•	•	•				0.52
3. Provider adherence to  intervention	•	•	•		•					0.41
4. Non-specific treatment  effects	•		•							0.24
5. Treatment manual used		•	•			•	•			0.34
Receipt of treatment
1. Participant  comprehension  assessed	•		•	•						0.52
2. Method for improving  comprehension	•	•	•	•	•	•	•	•	•	0.79
3. Performance skills  assessed	•	•	•	•	•	•				0.59
4. Strategy to improve  performance	•	•	•	•	•	•	•			0.66
Enactment of treatment skills
1. Skills assessed in  relevant settings	•	•	•	•	•		•		•	0.93
2. Improve performance in  relevant settings	•	•	•	•	•	•	•			0.79
Percent of fidelity elements described	.76	.84	.92	.64	.88	.76	.68	.28	.42
Study statistically significant		•		•	•	•	•

Note. A dot [•] indicates that the fidelity characteristic was present or that the study findings were statistically significant.

Evaluation of treatment fidelity can help to guide future research in several ways. A perfectly designed study will fail to find valid or significant results if the design is not adhered to. The purpose of the treatment fidelity guidelines is to codify whether a study has followed its design protocol with respect to the proposed intervention. Also included in the treatment fidelity is treatment receipt (whether the study participant has received the treatment), treatment enactment (whether the study participant applies what was learned in everyday life), and training of treatment providers (Borrelli et al., 2005). At the center of treatment fidelity is the delivery of treatment: having a method to ensure the content and dose is delivered, assessment of provider’s adherence to treatment protocol, and the use of a treatment manual.

The current review of SHS exposure interventions among children aims to: (1) evaluate treatment fidelity within the extant literature concerning SHS reduction interventions targeting children; (2) examine the relationship of treatment fidelity to obtaining statistically significant findings, year of the investigation, domestic vs. international investigations, intensity of the investigation, having a theoretical basis, and effectiveness vs. efficacy trials; (3) compare treatment fidelity ratings obtained in the present investigation to those obtained in previous treatment fidelity ratings of behavioral change interventions in general; (4) consider improvements to the treatment fidelity evaluation procedure; and (5) discuss the effects of theoretical fidelity, measurement fidelity, measurement reactivity, and SHS contamination in effecting the outcomes of SHS reduction interventions targeting children in their home.

Methods

A total of 29 studies were evaluated in terms of treatment fidelity according to the guidelines provided by Borrelli et al. (2005). The absence or presence of 26 treatment fidelity attributes was evaluated for each study and percent scores were developed. A proportion score was calculated across attributes and within study and could vary from 0 to 1.00 where 1.00 indicated that a study had fulfilled each of the 26 fidelity attributes. A second proportion score was calculated across studies and within fidelity attribute; it also could vary from 0 to 1.00. A score of 1.00 indicated that all studies had met the highest fidelity standard for a particular attribute. The scores were compared to study outcome (i.e., statistical significance), year of the investigation, domestic vs. international investigations, intensity of investigation, and efficacy vs. effectiveness trials. Statistical significance of the outcome of each study was determined independent of its fidelity rating.

Higher treatment fidelity was hypothesized to be associated with obtaining statistically significant findings, domestic as compared to international investigations, more recent as compared to older investigations (operationalized generally as year 2000 and earlier vs. Hovell, Zakarian, Matt, et al., (2000) and later—see below), efficacy as compared to effectiveness investigations, having a theoretical basis, and higher as compared to lower intensity interventions. Intervention intensity was operationalized in this way: brief interventions were those with one or two limited (typically 5 minutes or less) contacts with participants; if counseling was included it was minimal and generic. At least moderately intensive interventions were those with typically more than two contacts of longer duration (15–30 min) and included more extensive counseling or coaching that was tailored specifically to the study participant. Univariate analyses using either chisuare or the Mann-Whitney U Test were used to determine the relationship between fidelity and study outcome, location of study, recency of investigation, efficacy vs. effectiveness study, having a theoretical basis, and intervention intensity. A logistic regression to predict statistical significance (yes-no) was estimated with treatment fidelity (as a continuous variable), domestic vs. international investigations (yes-no), year of investigation (pre vs. 2000 or later), efficacy vs. effectiveness investigation, and intensity of investigation (brief vs. at least moderately intensive investigation), having a theoretical basis (yes-no) as predictors.

Treatment fidelity

Disparities in results across SHS intervention studies may be due in part to variability in treatment fidelity. To date, existing reviews of SHS interventions among children have not examined the issue of treatment fidelity. Following the criteria developed by Borrelli et al. (2005) (see Table 1), the 29 studies were evaluated in terms of 26 treatment fidelity attributes. The five areas of treatment fidelity included: treatment design, provider training, delivery of treatment, receipt of treatment, and enactment of treatment skills. Within each of these five areas were a number of subcategories. As stated in the 2005 Borrelli et al. article, a coding form was available, and this was provided for use in the present analysis. The published coding form (Borrelli et al.) differed from the one provided in that the latter included one additional item: “If more than one intervention condition is described, are they all described equally well?” This additional item was included in evaluation of the 29 articles.

Table 1.

Elements of treatment fidelity and average ratings by type of intervention.

	Intervention
	Behavioral change (1990-2000) N=342	SHS reduction (1987-2006) N=29
Treatment design
1. Provided information about treatment dose in the intervention condition
Length of contact session(s)	0.63	0.79
Number of contacts	0.86	1.00
Content of treatment	0.94	1.00
Duration of contact over time	0.92	1.00
2 If more than one intervention is described, all are described equally well	—	—
3. Provided information about treatment dose in the comparison condition
Length of contact session(s)	0.64	0.25
Number of contacts	0.83	0.96
Content of treatment	0.90	0.93
Duration of contact over time	0.87	1.00
4. Mention of provider credentials		0.52
5. Mention of a theoretical model or clinical guidelines on which the  intervention is based	0.71	0.72
Training providers
1. Description of how providers were trained	0.25	0.45
2. Standardized provider training	0.25	0.48
3. Measured provider skill acquisition post-training	0.16	0.28
4. Described how provider skills maintained over time	0.22	0.38
Delivery of treatment
1. Included method to ensure that the content of the intervention was being  delivered as specified (e.g., treatment manual, checklist, computer  program)	0.46	0.62
2. Included method to ensure that the dose of the intervention was being  delivered as specified (e.g., records number of contact minutes)	0.31	0.52
3. Included mechanism to assess if the provider actually adhered to the  intervention plan (e.g., audiotape, observation, self-report of provider)	0.27	0.41
4. Assessed nonspecific treatment effects such as warmth or credibility of  a counselor	0.06	0.24
5. Used a treatment manual	0.35	0.34
Receipt of treatment
1. Assessed subject comprehension of the intervention during the  intervention period	0.40	0.52
2. Included a strategy to improve subject comprehension of the interven-  tion above and beyond what is included in the intervention	0.52	0.79
3. Assessed subject’s ability to perform the intervention skills during the  intervention period	0.50	0.59
4. Included a strategy to improve subject performance of intervention skills  during the intervention period	0.53	0.69
Enactment of treatment skills
1. Assessed subject performance of the intervention skills assessed in  settings in which the intervention might be applied	0.69	0.93
2. Assessed strategy to improve subject performance of the intervention  skills in settings in which the intervention might be applied	0.46	0.82

Note. Reported average treatment fidelity ratings are from the Borrelli et al. (2005) review of general behavior change interventions and the present review of SHS interventions.

The authors used the following method for evaluating each of the 26 treatment fidelity criteria. Two authors (Johnson-Kozlow and Sirikulvadhana) independently rated each study and assigned “present” or “absent” for each of the coded elements. Any differences in ratings were resolved by follow-up discussion between the two coders. A third author (Rovniak) independently rated 6 (21%) of the 29 articles. Prior to the independent ratings, the first author met with the independent coder to go over interpretation of the coding form but without reference to the randomly selected articles.

Articles were selected using a block randomization method. The six articles were selected in this way: three were selected from pre-Hovell, Zakarian, Matt, et al., (2000) (14 articles), and another three from Hovell, Hillman, et al. (2000) and later articles (15 articles). This method was done because earlier articles addressed fewer fidelity issues than later articles. The Hovell et al. article published in 2000 was selected as the cut off because it demarcated equal numbers of articles in each group and because compared to previous articles the Hovell et al. investigation had exceptionally high treatment fidelity (0.92), thus setting a standard for reporting against which following investigations might refer. The three articles selected within each year group were based on fidelity rating, with fidelity categorized into low, moderate, and high levels using lower, middle, and upper thirds of the fidelity distribution. The use of the evaluation form on a wide range of fidelity was desirable, as difficulty in assigning ratings varied by fidelity level (articles with higher fidelity were easier to code than those with lower fidelity).

Following methods by Borrelli et al. (2005), percent agreement between the original ratings and the six ratings for the randomly selected studies was calculated. Percent agreement was calculated as the number of matching ratings divided by the number of applicable items for each article, and could vary from 1 to 1.0, with higher numbers indicating greater agreement. The percent agreement between originally coded and the randomly sampled articles was 0.78 and varied from 0.64 to 0.92 across the six interventions. The treatment fidelity codes assigned to all of the 29 SHS interventions are found in Table 2.

Results

Statistical significance

The higher the fidelity, the more likely the study was to report a significant outcome for the intervention (Mann-Whitney U Test, p=.003). The average fidelity for statistically significant studies was 0.76 compared with 0.50 for nonsignificant studies. Studies with high fidelity (based on a median split) were nine times more likely to obtain a significant result.

Year of investigation

Higher fidelity was associated with studies published after year 2000 (p=.020) based on the Mann-Whitney Test. Fidelity of the later studies was 0.74 compared to .56 for the earlier studies.

Domestic vs. international investigations

Domestic investigations had higher treatment fidelity than international investigations (p=.028) based on the Mann-Whitney U Test. The treatment fidelity for domestic investigations was 0.72 compared with 0.54 for international investigations.

Efficacy vs. effectiveness interventions

As hypothesized, efficacy studies, which are designed to take place under optimal conditions, were 6.9 times more likely to be judged to have high fidelity (p=.016) compared to effectiveness studies.

Intensity

Based on chi-square analyses, brief investigations were 16.5 times more likely to have lower fidelity (based on median split) than moderately intensive or intensive investigations (p=.001).

Theoretical basis

Based on chi-square analyses, having a theoretical basis was three times more likely to have higher fidelity (based on median split) than not having a theoretical basis (p=.001).

Statistical significance after adjusting for other variables

The relationship between fidelity and statistical significance remained after adjustment for year of investigation, domestic vs. international investigations, efficacy vs. effectiveness investigations, having a theoretical basis, and intensity of the investigation. Only fidelity was significantly related (p=.052) to observing statistically significant results; all other predictors were nonsignificant (p>.10).

Comparison with previous ratings

Ratings obtained from the present study were compared with those obtained from the previous Borrelli and et al. (2005) investigation that evaluated 342 behavior change interventions published in five journals over 10 years from 1990 to 2000. The purpose of this analysis was to determine if there were any differences in the ratings. Notable differences in rankings might indicate difficulties in using the evaluation guidelines by subsequent users compared with the original developers of the guidelines. Alternatively, it might indicate differences in treatment fidelity between SHS reduction interventions and behavioral change interventions overall.

As mentioned earlier, one of the components on which SHS intervention articles were rated was not used in the previous article—this was equal description of multiple control or comparison groups. This item was ignored in the following analyses. Ratings in the previous review ranged from a low of 0.06 to a high of 0.94; ratings in the present investigation ranged from 0.21 to 1.00 (see Table 1).

The treatment fidelity ratings for the two studies were rank-ordered and Spearman’s rho was calculated. The rank-order correlation between the ratings was 0.84. However, the median rankings in the previous investigation were lower than those in the present investigation (0.52 vs. 0.62). Using a two-way mixed analysis of variance, absolute agreement of non-averaged rating resulted in an intra-class correlation between the ratings of 0.79. The lower intra-class correlation in comparison with the rank-order correlation reflects the absolute differences (0.62 vs. 0.52) in codings between the current and earlier investigation.

Limitations of the evaluation guidelines

Methods vs. results

Zakarian et al. (2004) was assigned a treatment fidelity score of 0.92, yet no statistically significant results were observed. Zakarian et al. pointed out the difficulty in delivering an intervention in a clinical setting, including significant turnover among clinic staff, inadequate time at work to complete the study protocol, and lack of monetary compensation for their efforts. These results suggest that even with strong measurement protocols, substantial effort to maintain regular intervention contact with study participants, and efforts to tailor the intervention to individual participant needs, limitations in treatment fidelity may compromise intervention outcomes. These observations may point out another limitation of the evaluation guidelines which would have been expected to result in a lower treatment fidelity rating. The guideline asks whether there was a method to ensure that the dose was delivered. There were methods in place; however, this does not mean that it was possible to deliver the dose as specified or that other elements of treatment fidelity actually occurred.

Presence vs. absence

Another concern with the guidelines is the reliance on presence or absence of each attribute, as opposed to the use of a scale that indicates the degree of meeting a guideline. For example, the quality of provider training could be measured on a 5-point scale. Based on the present review, there were significant differences in the quality of the training, thus suggesting that a scale might be more appropriate than a yes-no check.

Equal weighting

The 26 fidelity attributes are given equal weight in their use to assess treatment fidelity overall. However, some attributes may be more important than others in determining treatment effects and therefore should be weighted relatively greater for fidelity assessments. For example, in many interventions, a strong theoretical foundation may be more important than non-specific treatment effects, such as warmth of the treatment provider. In addition, some attributes, such as equal description of more than one treatment condition, would not be applicable in all studies. Thus, this attribute would not be relevant in overall treatment fidelity in most cases. In the present investigation, this attribute was relevant in 5 of the 29 studies reviewed (17%).

Multi-cultural emphasis

Although the guidelines measure whether study participants receive the intervention (and this might be affected by comprehension of materials among those for whom English is a second language), it seems that more attention could be paid to multi-cultural issues in the guidelines and the coding form in particular. For example, an explicit category that acknowledges appropriateness of materials for non-White or immigrant populations is suggested for inclusion. Such materials go beyond translation into a different language to embody and communicate cultural practices of the target population.

Efficacy vs. effectiveness

The purpose of an efficacy study, which represents the majority of studies in the present investigation, is to establish the efficacy of an intervention under ideal conditions. The purpose of an effectiveness study, of which there are some in the present investigation, is to establish the effectiveness of an intervention in real-world, less-than-optimal conditions. Effectiveness studies often result in null outcomes in comparison to efficacy studies due to failures in implementation of the treatment plan. This may suggest that different guidelines be applied to efficacy vs. effectiveness investigations. While the efficacy guidelines might follow the current format, the effectiveness guidelines might include additional fidelity elements. These might include willingness of provider staff to undertake the intervention, staff turnover, funding and time allotted to staff to conduct the study, and support of study by decision-makers in the applied setting. The addition of such measures would provide more diagnostic and potentially prescriptive information for improving the fidelity of future effectiveness studies.

Application to ecological studies

A report from the Institute of Medicine (2001) investigating ways to modify health behavior concluded that the most effective interventions used an ecological approach. Similarly, the Task Force on Community Preventive Services, implemented in 1996 by the Centers for Disease Control and Prevention, provided a systematic review of the effectiveness of health behavior change methodologies (Hopkins et al., 2001). The task force determined that an environmental approach was effective in reducing exposure to SHS and based on their review, recommended that this strategy be implemented on the basis of its strong evidence of effectiveness. The current treatment fidelity guidelines are more applicable to clinical, one-on-one treatment interventions, which are the most commonly investigated behavioral interventions. With the current emphasis on ecological and multilevel investigations, new treatment fidelity guidelines are warranted. In time, these will be especially important as theory changes from pre-dominately introspective toward ecological models of health behavior.

Theoretical fidelity

Currently, the evaluation guidelines determine whether a theory was used or was not used, not the degree of following the theory—this is what has been termed theoretical fidelity (Rovniak, Hovell, Wojcik, Winett, & Martinez-Donate, 2005). Theories differ in their complexity and breadth; take for instance, differences between the Health Belief Model vs. the Social Cognitive Model. Thus, it is recommended that theoretical fidelity be considered as well as treatment fidelity. This would allow researchers to examine the merits of different theories in behavioral change interventions. The degree of adherence to theoretical fidelity is recommended, not merely its presence or absence.

Specifically, for this paper, theoretical fidelity is the extent to which intervention procedures match the theories upon which they are based (Rovniak et al. 2005). Previous research suggests that interventions with greater theoretical fidelity, or precision in following theory-based recommendations, have stronger effects (presumably due to valid theory) and provide more accurate tests of theory or means by which a given theory can be refined. Rovniak et al. compared high- and low-theoretical fidelity conditions in an e-mail-based physical activity intervention among 61 sedentary women. Both conditions received a full 12-week walking program and had equivalent treatment contact. However, the high-fidelity condition more precisely followed theory-based recommendations for implementing goal setting, self monitoring and feedback than the low-fidelity condition. The high-fidelity condition resulted in significantly faster walking time, as the intervention intended, and greater program satisfaction.

One investigation, by Zakarian et al. (2004), can be evaluated in terms of theoretical fidelity as well as other methodological issues in SHS reduction interventions and will serve as an exemplar. In that study, counseling was delivered by trained clinic staff as part of well-baby health services in a community clinic setting to decrease child’s exposure to SHS. The intervention was delivered to 76 mothers, as compared to 74 mothers in the measures-only control group. The 6-month intervention was based on the Behavioral Ecological Model (BEM) (Hovell, Zakarian, Matt, et al., (2000); Hovell, Wahlgren, & Gehrman, 2002) with emphasis on learning theory principles, including contingency contracting, self-monitoring, problem solving, and contingent reinforcement. Parent-reported child SHS exposure and the child’s urine cotinine were measured at baseline and at 3, 6, and 12 month follow-up. Parent-reported SHS exposure indicated a steep decline in exposure for both groups from baseline to 6 months, and remained level until month 12. Children’s urinary cotinine showed a nonsignificant decrease in SHS exposure in both group (see Figures 1 and 2).

Figure 1.

Children’s reported exposure to mother’s exposure to cigarettes in the home (cigarettes per week), by self-report. From Zakarian et al. (2004). Used by permission.

Figure 2.

Children’s urinary cotinine concentration (ng/ mL). From Zakarian et al. (2004). Used by permission.

Zakarian et al. (2004) included key constructs of the BEM in the counseling protocol, including modeling, feedback, and reinforcement. However, the investigation did not result in significant differences between the intervention and control group at 3, 6, and 12 month follow-up. The failure of the intervention to yield positive outcomes may reflect a lack of theoretical fidelity in this study. Although the theory was well-elucidated, the trial did not systematically track whether the BEM theoretical concepts were included in each counseling session, or whether the manner in which these constructs were implemented was consistent with the theory. For example, to implement modeling (a key construct in the BEM) with high theoretical fidelity, the principles for implementing modeling effectively would first need to be outlined, and then implemented consistently for each participant. These principles include breaking up complex activities into doable steps, having the study participant practice the activity, giving corrective feedback in an iterative fashion until mastery is attained, and incorporating multiple models of the behavior at diverse levels of the participant’s social network. Further, to generate an environmental contingency among the participant’s family, friends, co-workers and others, a more thorough protocol for incorporating interacting contingencies would have been needed. Difficulties implementing these theoretical tenets are not uncommon; however, there has been little discussion of these difficulties or potential solutions in previously published SHS reduction interventions with children (Rovniak et al., 2005). Thus, limited outcomes in this example study might be attributed to testing an incomplete intervention, based on the Behavioral Ecological Model.

Measurement fidelity

Fidelity is not limited to treatment or theory, but also includes all other aspects of design. Measures should be implemented as intended, and provide reliable and valid assessment of key treatment processes, theoretical constructs, and treatment outcomes. In SHS intervention studies, there are also challenges to achieving high-quality measurement that extend beyond staff protocols, including SHS contamination and measurement reactivity.

SHS contamination

Nicotine and other compounds from smoking adsorbs onto surfaces (as opposed to being absorbed into the surface) with which it comes in contact (Matt et al., 2004). This includes hard surfaces, such as counter tops, as well as carpets and drapes. However, once nicotine contamination has taken place in a home, it may also contribute to SHS exposure, even if no one is smoking in the home at the time. This may take place when nicotine “off-gasses” inside the home. Thus, biomarker assays of SHS, such as cotinine, include exposure to active smoke that occurs in the same room as the smoker, exposure to the dispersed smoke from a smoker in another part of the home, and exposure from off-gassing of nicotine at times when no one is smoking (USDHHS, 1986; USDHHS, 2006). Such processes are true in cars as well, and may have greater health impact due to the limited size of the enclosed space in the family vehicle, and while driving, a high prevalence of exposure to children (Emerson et al., 1994).

Because such contamination may persist in the environment for several months (Matt et al., 2004), these findings present difficulties for measuring an individual’s current exposure to SHS and for evaluating interventions to decrease exposure when using cotinine as the primary outcome measure. Matt et al. (2004) demonstrated the effect on infants under age 12 months of off-gassing in homes of smokers. SHS contamination was measured by nicotine in household dust, indoor air, and household surfaces; SHS exposure was assessed by urine cotinine. Based on infant’s urine cotinine, SHS exposure was 5 to 7 times higher in households of smokers who smoked outdoors than in households with nonsmoking parents. The increased SHS exposure may have been due to contamination of the parent’s clothes, skin, and hair that is brought into the living environment, outdoor smoke that drifts in through an open window, or old contamination of the home that occurred in the distant past.

To identify the possible effects of SHS contamination, both self-report and cotinine SHS exposure data must be collected and reported at two or more time points; the 2004 Zakarian article has these measures. If SHS contamination was a factor in an SHS reduction investigation we would expect less effect according to cotinine but a greater effect observed by self-report measure. This was the case for the Zakarian study (see Figures 1 and 2). There was a large, statistically significant decrease in reported number of cigarettes to which the child was exposed over time, from a geometric mean of about 54 cigarettes per week at baseline to about 20 cigarettes per week at 6 months after the intervention ended (a 63% reduction). However, the geometric mean urine cotinine decreased only 26% during the same interval, from 9.39 to 6.93 ng per ml; this resulted in a statistically nonsignificant decrease over time.

These results suggest that even in the presence of a 63% decrease in indoor exposure over a 1-year period, biological exposure may decrease only 26%, possibly due to persistent residual contamination in the child’s environment. This suggests that the child continued to be exposed to SHS but this was not assessed by the self-report measure. It may also suggest that to demonstrate a greater decrease in cotinine in the experimental vs. control condition, significant mitigation efforts are needed to reduce contamination, such as never allowing any smoking in the home (or car), and that additional time is necessary to achieve sufficient decrease in the off-gassing over time in order to be detected in biomarkers such as cotinine.

Measurement reactivity

Another issue to deal with in SHS intervention studies is measurement reactivity (Hovell, Zakarian, Wahlgren, et al., 2000). Reactive measurement refers to changes in behavior that are due solely to measurement processes, and occur independently of the intervention under investigation. The reactive effects of an interview in the context of an SHS counseling study are the result of several processes (Webb, Campbell, Schwartz, & Sechrest, 2000). Participants under scrutiny by researchers may have increased awareness of their SHS-related performance. The skills of the interviewers may affect the extent to which they establish a relationship with the participant that inadvertently encourages cooperation, trust, and self-disclosure, and possibly change in tobacco use to “please the interviewer.” Measurement thus serves as a change agent by providing feedback about the behavior being measured. Interviews may stimulate interest in a particular topic, provide information about norms, standards and goals, give feedback about one’s behavior, direct attention to a particular behavior and its consequences, and imply strategies for modifying the behavior. Subsequent change may be reinforced by the feedback process inherent in the repeated measurement process. Similarly, the presence of air monitors in the home and giving urine samples, as is common in SHS reduction interventions, are likely to result in measurement reactivity.

Thus, reactivity may play an important role in changing health-related behavior that is independent of the experimentally-controlled intervention. It may affect only the control group appreciably, diminishing the potential effect size achievable by the intervention, or it may also affect the group assigned to receive the intervention, by enhancing its effectiveness.

Nevertheless, it is important to address several competing explanations for decreases in SHS exposure in both intervention and control groups as has been found in several previous studies. The decline could reflect an historical trend that affects all participants regardless of group assignment. This is not likely, however, because reduction in SHS has been observed in different years, using different target groups, and utilizing different methodologies, including sequential recruitment over at least a year. Thus, secular trends or the confounding effects of an ongoing community-wide intervention are unlikely explanations for changes in exposure observed across participants at a given point in the study (e.g., the post-test measure). Regression toward the mean could produce SHS reduction in the control group if participants were selected based on extreme exposure. However, this selection strategy was not used in previous SHS reduction intervention studies that demonstrated measurement reactivity.

Another explanation assumes a change in the validity of parent-reported data over time. For instance, participants may over-estimate their exposure at baseline and then revise their estimate in followup assessments. This may occur because the longer that participants, either control or intervention, are in the study, the more likely they are to be sensitized to the social undesirability of exposing their child to passive smoke. However, investigations of the validity of parent-reported SHS at baseline and post-treatment, evaluated as correlations with urine cotinine and air nicotine, reveal comparable validity correlations in both treatment and control groups over time (Matt et al., 2000), making it unlikely that the decline in mean levels of self-reported exposure can be accounted for by a change in the validity of parent reports.

Previous research (Hovell, Meltzer, et al., 1994; Wahlgren, Hovell, Meltzer, Hofstetter, & Zakarian, 1997) sought to formally disentangle the effects of measurement reactivity from treatment effects. These studies (the original 12-month investigation and the 2-year follow up study) compared SHS exposure reduction in two control groups (one with minimal and the other with complete measurement) with a condition with counseling plus full measurement. All three conditions showed a decrease in SHS reports during the first two repeated baseline measures, before any counseling was started. The level of decrease over 12 months was least for the limited measures control group, somewhat more change was observed in the control group with full measurement, and the greatest change was observed for the full measurement plus counseling condition (Hovell, Meltzer, et al.). The magnitude of decrease by the second baseline approximated the magnitude of decrease seen in the full measurement and counseling condition, suggesting that measurement reactivity could yield therapeutic change of similar magnitude as that achieved by clinical interventions. Further, from 20- to 30-months post-baseline those in the full measurement control group continued their decline in child’s SHS exposure while those in the minimal measurement control group showed a slight increase in exposure (Wahlgren et al., 1997). These studies greatly increase the plausibility that changes in both efficacy and effectiveness trials are due, in part, to measurement reactivity. This process deserves more formal study and the measurement process as “intervention” deserves more consideration.

Measurement reactivity may explain why the Zakarian (2004) investigation reported decreases in child’s exposure to SHS in both intervention and control groups over time. With respect to self-report measures, the dramatic decline in number of cigarettes to which the child was exposed from baseline to post-intervention cut exposure in half in both intervention and control groups.

Discussion

Previous reviews of SHS-reduction interventions in children have concluded that more intensive interventions yield better results (Gehrman & Hovell, 2003); we observed similar results. Based on Chisquare analyses, brief investigations were 16.5 times more likely to have lower fidelity (based on median split) than moderately intensive or intensive investigations (p=.001). However, even with more intensive interventions SHS exposure may not decrease, or if it does decrease, treatment effects typically wane post-intervention. Little research has explored alternative factors that may contribute to the effectiveness of SHS interventions. Increasingly, researchers emphasize the importance of assessing the quality of intervention procedures and measures (Bellg et al., 2004; Borrelli et al., 2005; Calsyn, 2000; Rovniak et al., 2005).

The present review discussed the influence of treatment fidelity on treatment outcomes across 29 SHS reduction interventions that targeted children. The results indicated that studies with higher treatment fidelity were more likely to obtain statistically significant results. Univariate analyses serve as evidence of construct validity of the treatment fidelity measure: fidelity was associated with more intensive domestic efficacy investigations using a theoretical basis that had been conducted in recent years. After adjustment for these variables only fidelity was significantly related to obtaining statistically significant results. These results suggest that the treatment fidelity guidelines are valid and should be used to inform intervention design and reporting of behavioral change interventions.

The wide applicability of the guidelines is supported by a large correlation (0.84) between the relative use of the attributes in 342 behavioral change interventions in comparison to the 29 studies of the present investigation. The somewhat lower intraclass correlation of 0.79 reflected the use of higher fidelity ratings from coders in the present investigation relative to the earlier Borrelli review (2005). Obtaining higher fidelity ratings might indicate that the coders in the present investigation were more lenient or that there were real differences in fidelity between SHS reduction interventions vs. behavioral change interventions overall. It would also be expected that the larger the number of investigations, the lower the fidelity average, as was observed, thus increasing the difference in ratings between the earlier vs. the current ratings.

Raters in the present investigation encountered some difficulties in using the coding form, which was reflected in the moderate percent agreement of 0.78 between independent raters. Additional training using more explicit directions in the use of the coding form, and more objective evaluation criteria may be warranted. Nevertheless, explicit inclusion of the treatment fidelity guidelines in SHS reduction interventions might help jump start SHS reduction research targeting children in the U.S., which, in recent years has been comprised mainly of international investigations.

Some areas that needed particular emphasis in future SHS reduction studies include use of a treatment manual, assessment of provider skills, degree of provider adherence to the intervention protocol, and a description of how providers were trained. Each of these treatment fidelity attributes were significant predictors of treatment fidelity overall but were used in 50% or fewer of the studies (data not shown). Length of contact in the control or comparison condition, which was described by only 25% of the 29 studies under review, might also be emphasized. In the context of measurement reactivity, the longer the contact with the control condition, the more likely participants would be predicted to modify their health behavior.

Several adjustments or modifications to the guidelines were suggested. These included the use of evaluative scales when appropriate in lieu of the yes-no categories, weighting schemes that acknowledge the importance of some treatment fidelity elements over others, considering not only whether a fidelity issue was assessed but the outcome of the assessment, the inclusion of adherence to methods which take into account multi-cultural practices, and additional fidelity criteria for effectiveness trials or ecological studies. Potential problems associated with SHS contamination and measurement reactivity on intervention results were also discussed.

The 2004 Zakarian et al. investigation served as an example from which to discuss several fidelity and measurement issues which may have led to null outcomes. First, Zakarian et al. acknowledged difficulty in attaining treatment fidelity, and may also have had poor theoretical fidelity, thus null outcomes were more likely to be observed. However, it also appeared that SHS contamination might have been present, as the self-report data (which were observed to be both reliable and valid) demonstrated a sharp decrease in exposure while cotinine results remained close to level. This suggests that the child continued to be exposed to SHS through contamination in the home. Second, measurement reactivity appeared to be very strong in the Zakarian et al. investigation. According to self-report measures, both the control and the intervention groups showed a sharp decline in child’s exposure to SHS that maintained over time. However, the Zakarian et al. study may have suffered from incomplete adherence to the BEM. Since contingency management procedures were emphasized with less attention to ecological context, this study might be considered compromised in its adherence to the theoretical model.

Ultimately, the failure to obtain effects in a study based on a given theoretical model, in the context of high theoretical and methodological fidelity, implies an incomplete, inaccurate, or false theory. The use of theoretical fidelity standards in the design and conduct of new studies, and in the critique of completed studies, should help cull out the weak, underspecified and inaccurate theories. Thus, in our call for use of fidelity standards and procedures, we are suggesting a means of more formally developing valid theory. This also suggests that future interventions should be based on theories or theoretical constructs that demonstrate behavior change, and that studies should be designed that contrast promising theories to determine their relative merits.

Though studies vary in their methodology, most have reported decreases in SHS or smoking levels and many also have found decreases in the control condition. The consistency of the decreases in both control and experimental conditions appears to be best explained by reactivity to measurement. This, in turn, suggests that studies need to be more powerful to detect intervention effects and that researchers should include formal feedback procedures to maximize the power of their interventions. The latter will require more creative designs to test combined feedback and counseling procedures.

SHS reduction interventions typically include objective measures of SHS exposure that are used to validate self-reported exposure. However, evidence suggests that cotinine may reflect not only current exposure, but on-going exposure to contaminated microenvironments. If this is true, cotinine may not provide the best outcome measure for short-term effects of interventions that are designed to minimize the child’s presence when smoking occurs. Thus, such interventions should use reported measures of change that have been validated by cotinine assays that are based on the same time period as measured by reports.

Future intervention studies should be designed to effect change in children’s presence when others are smoking, to eliminate smoking within the home by all smokers and at all times, and to decontaminate the home and car. In order to allow sufficient time to assess the effect of decontamination, the intervention and follow-up periods should also be longer. Without these improvements, SHS reduction interventions would not meet ideal methodological fidelity standards, especially for protecting the health of children. Moreover, to design an intervention that protects children from visible smoke and at the same time from off-gassing or contaminated environments will require a much broader intervention aimed at changing different kinds of behavior for a larger group of people than mothers or one caretaker. Assessing clinical interventions, such as counseling, or even ecological interventions such as the effects of a state-wide tobacco control program in concert with counseling may be best evaluated with changes in visible smoke exposure. However, public health outcomes demand more complete interventions that may involve changing many behaviors of many different players in the community to prevent contaminated homes or decontaminate those already contaminated by nicotine and other toxic constituents of tobacco smoke.

The literature remains shallow with regard to investigation of multi-level and ecological influences as well as moderating functions of environmental and social contexts on SHS exposure of children. However, sufficient theory is available to test moderating and mediating functions of social contexts and tobacco policies in intervention studies. As these more complex designs are adopted, results will both inform existing theory and help to select the more valid theory among many. This will be accomplished by better understanding the distal effects of social and other environmental main effects on SHS exposure as well as the processes by which counseling and other clinical interventions are modified by ecological contexts. This direction of research will place behavioral studies, such as SHS exposure prevention and control interventions, more centrally in a public health model that does not exclusively emphasize clinical interventions as an explanation of, or means to, control lifestyle-mediated morbidity and premature mortality. This direction of applied science offers a more parsimonious way to understand and control tobacco use as well as consequential diseases.