Differences in Perceived Implementation of a Standard Versus Peer-led Interactive Substance Abuse Prevention Program

Abstract

Objective

To assess perceived implementation of 2 substance-abuse prevention programs: a standard one and a peer-led interactive one.

Methods

Data from 16 health educators were collected after 504 classroom sessions, 63 of which were observed by 24 monitors.

Results

In the interactive program, health educators (HEs) followed the curriculum less closely, reported less favorable classroom processes and less off-task talking than in the standard one.

Conclusions

These data indicate that an interactive substance-abuse prevention program does not necessarily entail more off-task discussion but also does not necessarily guarantee more favorable program implementation.

Project Towards No Drug Abuse (TND) is a school-based curriculum designed to motivate youth to change their perspectives on and perceptions of drug use, learn social and life skills to bond to prosocial institutions, and learn decision making to help them plan good solutions to complex, problem situations.¹ A one-year follow-up study of TND has shown decreased use of cigarettes, alcohol, marijuana, hard drugs, and weapons carrying.² Effects on cigarette smoking, hard drug use, and marijuana use (the latter, among male nonusers at baseline) were also maintained at 2-year follow-up. ³

TND, and many other substance abuse prevention programs, have been created and implemented with limited success perhaps because they have not adequately incorporated the power and influence of peer social networks. Social networks have been shown repeatedly to be a significant covariate of adolescent substance use.^4–7 Most programs, however, incorporate peer influence factors only tangentially, primarily through role modeling and resistance-skills training.^8–11

Our prior work with a tobacco prevention program showed that using network information to structure a school-based tobacco prevention program increased its effectiveness.^11,12 The statistical analysis revealed a program by network interaction such that the network condition, peers assigned to groups based on their social networks, was more effective in the culturally tailored curriculum than in the traditional social-influences one. Specifically, the culturally tailored program was effective in preventing smoking if students participated in the activities in groups composed of their social network members. However, a traditional social-influences program did not appear to be more effective when students participated in groups with their social network members.

Although these early experiences with network-based implementation strategies are promising, a question was raised whether network strategies can be used with existing evidence-based programs. The present study was designed to determine if the effects of TND could be enhanced by making the program more interactive, involving use of peer student leaders and more use of group work in the curriculum. The TND curriculum was modified by adding more interactivity to the sessions and, whenever possible, having all activities conducted in groups. We modified TND to allow for more interpersonal communication and group interaction in the hopes of increasing the processing of curricular materials. Group activities that were once directed by the health educator and involved the whole classroom were, in the networked version, topics for small group discussion led by trained group-selected peer leaders. For example, in session 3, small groups were asked to debunk a common myth about drug use such as “most of the students in my school smoke.” In the standard version this was presented to the whole classroom and then opened for discussion. Table 1 provides a summary of the TND curriculum changes to create the network version.

Table 1.

Summary of TND Curriculum Modifications

Session	Title	Modifications	Underlying Principle
A	Peer Leader Training	This session was added for students who were nominated by their class as the best leaders. Discuss with students what it means to be a good leader.	Strengthen the influence of the peer leader.
1	Active Listening	Active listening left the same. Process questions are in a group discussion format instead of instructor facilitated.	Strengthen the influence of the peer leader.
2	Stereo-typing	When completing the character lists, have students complete the list as a group. Have students brainstorm 2 additional positive character traits.	Create more group interaction. More processing time.
3	Myths & Denial	Peer leaders (PL) as classroom assistants. Each group discusses one of the 4 myths instead of as a classroom. PL summarizes discussion. Each group also discusses one of 4 types of denial instead of as a classroom. PL summarizes discussion.	PL models the appropriate response to the myths. More group interaction. Increased personal responsibility.
4	Chemical Dependency	Added a handout on chemical dependency and the family that PLs review with their group. Students instructed as a group to select items from the toolkit they feel are relevant. Instructed to share this with friends and family.	Increase social support. Increased process and group discussion.
5	Talk Show	—
6	Marijuana Panel	—
7	Tobacco Basketball & Use Cessation	Changed into 4 teams playing a game similar to “horse.” Teams ask each other questions from their cards. Additional questions were added. Points awarded for correct answers.Pages of the cessation manual laminated and each group receives a section. Groups are instructed to quickly select items from the bullets they feel are instructive.	Group cohesiveness increased. Group processing.
8	Stress, Health, & Goals	Group discussion added on how to handle stress. In groups students decide on which suggestions they think they could really use.	Group processing.
9	Self-control	Peer leaders model behaviors. Students asked to brainstorm ways that they can be supportive of other people’s positive behaviors.	Increase social support and processing of material
10	Positive & Negative Thought Loops	Added group discussions on how thoughts affect emotions and how violent situations can be avoided.	Processing time.
11	Perspectives	—
12	Decision Making & Commitment	Students brainstorm ways that drug abuse can negatively impact their lives. Peer leaders should support those willing to make healthy commitments.	Social support and additional processing.

TND Network added a session for peer leader training and encouraged peer leaders to take an active role in the implementation of the curriculum—a notable change from standard TND. Overall, the networked version relied less on the health educator and put more responsibility on the students to process and discuss complex topics. This difference in process could lead students to be relatively more likely to feel ownership and become involved in the responses they provide to questions in the curriculum because they had a relatively greater role in generating them. We anticipate several advantages of the TND Network program. Students who feel uncomfortable speaking in large groups may become more involved in the comparatively smaller groups created in TND Network.

Social network analysis^13–16 was used to identify peer leaders and construct groups. Students were asked to name other students in their class they thought would make good leaders for a classroom project. These data were then analyzed to identify the students who received the most nominations and thus selected as group leaders. Students who nominated these leaders were then assigned to be members in the corresponding group.¹¹ The purpose of the present study is to determine whether there were differences in the implementation of the 2 curricula and assess whether we can anticipate differences in outcomes based on implementation differences. Two sources of data for this process evaluation are health educator assessments conducted after each session they taught and observers who monitored selected sessions.

Health Educators’ Perceptions of the 2 Curricula

One of the main concerns in a study comparing curricula is that health educators (HEs) will be biased in their preference for one program over another.¹⁷ Thus a primary objective of this process evaluation was to determine whether the 2 curricula were implemented similarly as reported by HEs and observers. A second concern is that program delivery may be affected by HE characteristics, particularly HE experience.¹⁸ Teacher adoption of tobacco and other drug abuse prevention programs is positively associated with favorable attitudes toward the program, comfort with the program content and approach, perceived self-efficacy to implement the program, independence, innovativeness,¹⁹ a confident and nonauthoritarian teaching style, good overall teaching skills (which may take some experience teaching the program), and characteristics such as being outgoing, adventurous, and organized.^19,20 Techniques such as small groups, role playing, and peer leaders often require teachers to develop and apply new skills and change their student-teacher role relationship from one that emphasizes classroom control to one that is less predictable and more student centered.¹⁹ Comparison of curricula may be confounded if HE experience influences program delivery and different HEs implement the different curricula. Thus, a second aim is to determine if HE experience is associated with differences in program implementation.

The third focus was to determine, to the extent possible, whether there were differences in perceived program effectiveness as reported by HEs and observers. One of the goals of a process evaluation is to determine program effects during program delivery.^21,22 This is particularly important for interventions of longer duration because process variables are among those that may predict longer-term impact.^22,23 Thus, it is useful to know whether implementers and observers can report short-term effects and if they vary by curriculum.

A final aim of this study was to determine whether health educators or observers would notice more off-task discussion among the students in the network curriculum. Recent evidence has shown that deviant peers influence one another such that peer groups composed of deviant peers are more likely to engage in disruptive behavior than are groups composed of nonpeers.²⁴ The network curriculum deliberately puts students into groups based on their social network nominations; thus we expect increased discussion among the students in the networked condition, whereas interaction in the standard version is highly structured and health-educator controlled.²⁵ If too much of this discussion is not related to the curriculum, it could be disruptive rather than beneficial.

METHODS

The study was implemented in 14 southern California continuation high schools across 3 counties. We contacted 25 continuation high school districts in southern California to solicit participation. Of these, 17 did not participate for various reasons: 10 refused, citing administrative concerns; 7 were not used because the classroom populations were too small or some other restriction on access was placed. Of the 8 districts used for the study, 1 served as a pilot location and the remaining 7 provided classrooms that could be randomly assigned to one of 3 conditions: control (prevention as usual), TND, and TND Network.

TND and TND Network are both 12-session programs delivered over a 3- to 4- week period, predominantly with classes taught on Tuesday, Wednesday, and Thursday. Sixteen health educators were trained by program staff to teach TND and TND Network. Of these, 9 reported having no prior experience as a health educator, and 7 reported having some experience. Of the 7 with previous experience, 3 had previously taught TND or a similar program. Nine of the health educators were in the final semester or year of their master of public health degree and taught TND for course credit and a modest stipend. Twelve health educators taught both curricula, 2 HEs taught only TND, and 2 HEs taught only TND Network.

HE Assessment Data

At the conclusion of each session, health educators were instructed to complete a self-assessment form rating how much they modified the activities, whether they felt rushed while delivering the program, and their perceptions of session effectiveness (Appendix A). The assessment instrument was divided into 4 sections: (a) rate how closely they followed activities (items 1a – 1c); (b) report any changes that occurred during the class (items 2 – 8); (c) rate their perceptions of the classroom process (items 9–16); and (d) report whether students talked about things other than the program, whether any students stated their intentions to quit, and how well they thought the session was conducted (items 17–19). Sections 1 and 3 were 5-point Likert items asking health educators to state their level of agreement, and section 2 items were dichotomous (yes/no). Section 4 items were both Likert and dichotomous. We treated sections 1, 2, and 3 as scales and performed factor analysis; section 4 items were treated as individual items.

We attempted to have assessment forms completed by all 16 health educators for all 12 sessions delivered to the 47 classes, yielding a potential database of 564 assessments. Some assessment forms were missing: Specifically, forms were missing for 60 sessions across 7 schools and from 7 different health educators. The missing rate was the same between TND and TND Network and distributed across program sessions. There was one school and class for which all forms were missing (n=12) and another school for which forms were missing from 3 classes (n=36). Forms were missing usually due to health educator error in either misplacing the forms or forgetting to complete them. We compared rates of missing forms across health educators, schools, and sessions to determine if there was any bias in the missing data. One health educator did not return any forms (accounting for 2 classes and 24 assessments). Otherwise, no other health educator returned less than 75%, with 10 of 13 returning more than 90%. There was no bias in missing forms by session and no systematic pattern to the missing data other than the entire classes mentioned above. The final sample consisted of 504 assessment forms (88.1%). This is a census of all classrooms taught in the study.

Observer Monitoring Data

Health educators and program staff used a slightly reworded form to monitor program delivery. (For example, items were reworded to make logical sense; for example the HE assessment would ask, “To what extent did you elicit students’ responses?” and the monitoring form would ask, “To what extent did the HE elicit students’ responses?”) We attempted to observe all sessions 5 and 10 of the curricula, but due to scheduling conflicts observers sometimes observed other sessions. Of the 94 (47 classes by 2 sessions) potential observations, we completed 73 (77.7%). Some observations were missed due to scheduling conflicts, particularly when multiple classes were being taught at the same time.

Outcomes

We factor analyzed the monitoring and observer items in sections 1, 2, and 3 to determine if program delivery and process perceptions clustered within prespecified domains. We then construct perceived program delivery variables based on the factor analysis. Because the outcomes are not normally distributed, we dichotomized all variables on the mean within data (HE assessments and observers). We also constructed 3 outcome variables from the section 4 individual items.

Analysis Plan

Data analyses are complicated by dependencies in both data sets and by a desire to compare results between the data sets. For the health educator data, intraclass covariation (dependence) exists in the data in at least 2 ways: within-HE covariation is likely to be higher than between HEs; and within-session covariation is likely to be higher than between-session covariation (eg, HEs are more likely to report similar perceptions of lesson 1 than between lesson 1 and 2). Current statistical models do not allow for such nested covariance structures. In addition, in this study we want to assess the impact of health educator experience, and we cannot control for health educator covariation while testing for health educator experience. Consequently, in the analysis we control for intrasession covariation and do not measure health educator covariation. For the observer monitoring data, covariation exists within session, within HE, and within observer. Again, such multiple overlapping covariation structures cannot be modeled. For the observer data, we again control for intrasession effects and do not model individual health educator and observer effects.

Analyses consisted of nonpaired and paired t-tests comparing outcomes between curricula for the full sample and for the restricted sample for sessions in which we have both health educator and observer data. Analysis also consisted of logistic regression to determine whether outcomes were associated with curriculum (TND Standard vs TND Network) and health educator experience controlling for within-session covariation.

RESULTS

Scale Analyses

Confirmatory factor analyses for the HE assessment forms are reported in Table 2. Section 1 consisted of 3 items measuring how closely HEs followed the curriculum. Results show modest correlation among these items in both data sets. The psychometric properties (Eigen-values= 0.57, 1.31; alphas=0.42, 0.64) show that these items only marginally constitute a scale. The 3 items were averaged and then dichotomized on the mean to measure how closely the curriculum was followed.

Table 2.

Factor Analysis, Psychometrics, and Univariate Statistics for Process Outcomes

	Close		Changes		Process		Talk		Quit		Rate
	HE	Obs	HE	Obs	HE	Obs	HE	Obs	HE	Obs	HE	Obs
Q1a	0.43	0.14
Q1b	0.37	0.79
Q1c	0.51	0.81
Eigenvalue	0.57	1.31
Alpha	0.42	0.64
Q2			0.67	−0.23
Q3			0.14	0.51
Q4			0.21	0.05
Q5			0.11	−0.29
Q6			0.61	0.70
Q7			0.09	−0.11
Q8			0.26	0.32
Eigenvalue			0.97	1.00
Q9					0.70	0.75
Q10					0.69	0.72
Q11					0.26	0.68
Q12					0.59	0.45
Q13					0.58	0.62
Q14					0.49	0.39
Q15					0.41	0.12
Q16					0.18	0.51
Eigenvalue					2.14	2.56
Alpha					0.73	0.73
Mean	4.38	2.93	0.77	0.81	3.97	3.87	0.56	0.63	3.51	2.77	0.16	0.07
% in high Category	68.8	46.6	44.2	56.2	57.5	47.9	56.3	63.0	52.4	54.8	16.1	6.8

Note.

HE is health educator assessment data; Obs is observer monitoring data.

Section 2 consisted of 7 items measuring changes that occurred during implementation. Results show no correlation among these items and an inconsistent factor structure with different items loading on the first factor in the 2 data sets. Consequently we summed these 7 items and dichotomized on whether any changes were reported during implementation. Section 3 consisted of 8 items measuring classroom process. Results show a clear factor structure consistent across both data sets with good psychometric properties (Eigenvalues= 2.14, 2.56). Items 9, 10, 12, and 13 all loaded high on the first and only factor in both data sets and were combined to form a process scale (alphas=0.73, 0.73, TND and TND-Network, respectively).

Effects on Process Indicators

Table 3 compares outcomes by the 2 curricula for each data set. HEs were more likely to report following the TND curriculum more closely than in TND Network (0.72 vs 0.65, P<.10), but this difference was not supported by the observers. HEs and observers reported the same number of changes during program delivery. HEs and observers both reported better classroom processes for TND than TND-Network (0.62 vs 0.53, P<.10; 0.62 vs 0.37, P<.05). The overall rating of the session was the same by HEs, but observers rated TND sessions significantly better than TND-Network (0.81 vs 0.49, P<.01). HEs reported more off-task talking in TND than in TND-Network (0.57 vs 0.48, P<.05), but this difference was not supported by the observers. There were no differences between curricula on perceptions of program effects on quitting or cutting back (0.18 vs 0.15, TND and TND Network respectively, P=0.36).

Table 3.

Differences in Process Outcomes by Curriculum for HE Assessments and Observer Monitors

	HE Assessments (N=504)			Observer Monitors (N=73)
	TND	TND- Networked	Prob.	TND	TND- Networked	Prob.
Close	0.72	0.65	0.08	0.41	0.51	0.37
Changes	0.44	0.44	0.99	0.53	0.58	0.65
Process	0.62	0.53	0.06	0.62	0.37	0.03
Rate	0.57	0.56	0.79	0.81	0.49	0.01
Talk	0.57	0.48	0.05	0.5	0.58	0.47
Quit	0.18	0.15	0.36	0.06	0.07	0.86

Table 4 reports regression results for the 6 program implementation variables regressed on curriculum (TND vs TND Networked) and health educator experience for both data sets. In the HE assessment data, TND-Network was associated with less closely following the curriculum (AOR=0.71, 95% CI=0.55–0.92) and less favorable classroom processes (AOR=0.70, 95% CI=0.59–0.84) than was TND. The TND Network curriculum was associated with fewer reports of students talking about something other than the curriculum (AOR=0.65, 95% CI=0.48–0.89) than TND. Experienced HEs were more likely to report students talking about something other than the curriculum (AOR=2.49, 95% CI=1.76–3.53) and less likely to rate the session has having gone well (AOR=0.55, 0.40–0.75) than were inexperienced HEs.

Table 4.

Adjusted Odds Ratios for Associations Between Curriculum (TND Network) and Health Educator Experience on Process Outcomes

	Close	Changes	Process	Rate	Talk	Quit
HE Assessment Data (N=504)
TND-Networked	0.71 (0.55–0.92)	1.01 (0.68–1.49)	0.70 (0.59–0.84)	0.99 (0.77–1.27)	0.65 (0.48–0.89)	0.81 (0.51–1.29)
HE Experience	1.07 (0.67–1.71)	0.82 (0.62–1.08)	1.11 (0.81–1.53)	0.55 (0.40–0.75)	2.49 (1.76–3.53)	0.83 (0.48–1.44)
Observer Monitoring Data (N=73)
TND-Networked	1.47 (1.00–2.17)	1.16 (0.30–4.54)	0.25 (0.11–0.54)	0.17 (0.05–0.59)	1.4 (0.71–2.77)	1.16 (0.27–5.00)
HE Experience	3.01 (0.81–11.2)	4.19 (1.92–9.14)	6.59 (3.89–11.2)	5.49 (1.73–17.5)	1.08 (0.62–1.89)	1.48 (0.43–5.12)

In the observer monitoring data, TND-Network was associated with following the curriculum more closely (AOR=1.47, 95% CI=1.01–2.17), following less favorable classroom processes (AOR=0.25, 95% CI=0.11–0.54), and less likely to have been rated as having gone well (AOR=0.17, 0.05–0.59) than was TND. Experienced HEs were more likely to follow the curriculum closely (AOR=3.01, 95% CI=0.81–11.2), have changes during implementation (AOR=4.19, 95% CI=1.92–9.14), be rated as having more favorable classroom processes (AOR=6.59, 95% CI=3.89–11.2), and be more likely to be rated as having the session well conducted (AOR=5.49, 95% CI=1.73–17.5) than were inexperienced HEs.

We conducted some exploratory analyses by including the classroom process variable and an interaction term of curriculum and HE experience on regressions for the 3 dichotomous process outcomes: rate, talk, and quit. Results showed that process was strongly positively associated with all 3 outcomes. Specifically the adjusted odds ratios (AORs) in all 3 equations for process on outcomes in the HE assessment data were rate, 10.5 (95% CI=6.63–16.8); talk, 9.76 (95% CI=6.94–13.7); and quit, 2.09 (95% CI=1.25–3.49). For the observer monitoring data, the AORs were rate, 9.69 (95% CI=3.72–25.3); talk, 0.53 (95% CI=0.10–2.72); and quit, 133(95% CI=18.6–963).

Finally, of methodological interest to process evaluators is the agreement between HE self-reported assessments and scores on those same items by observers. Table 5 reports scores on the 6 outcomes for the session for which we have both HE assessments and observer data. (Sixty-three [86.3%]observer sessions also have data from the HEs.) HEs reported following the curriculum more closely than the observers reported (4.37 vs 3.0, P<.001), and HEs were more likely to report off-task talk (3.42 vs 2.72, P<.01). HEs and observers had similar ratings for the other 4 outcomes. Notably, the process scale scores were almost identical.

Table 5.

Comparison Between HE Assessments and Observers Monitoring of the Same Sessions

	HE Assessments (N=63)	Observer Monitors (N=63)	Prob- ability
Close	4.37	3.00	0
Changes	0.71	0.89	0.21
Process	3.86	3.87	0.92
Rate	0.58	0.65	0.43
Talk	3.42	2.72	0.01
Quit	0.09	0.06	0.32

DISCUSSION

Some differences in perceived program implementation were evident. HEs and observers both reported less favorable classroom processes (student interest, student understanding, classroom control, and HE confidence) for TND Network than for TND. This difference could be from the less controlled nature of TND Network, which had students working in groups. These additional group activities may have created an environment with less focused lesson objectives. It is also possible the group-oriented nature of TND Network created a situation in which students direct remarks to one another rather than the teacher so it is more difficult to rate student involvement and interest. At the same time, when students were talking with one another in TND Network, it may have been more likely to be about program content than other topics.

HE experience was an important influence on program implementation. Experienced HEs were observed to follow the curriculum as written, make more modifications during implementation to adapt to classroom factors (most likely to shorten or add activities), and had more favorable classroom process ratings than did inexperienced HEs. It may seem contradictory that more experienced HEs followed the curriculum more closely, yet also made more modifications. This stems from the definition of changes, which reflects changes made in response to external classroom factors (ie, class was shortened due to announcements or a fire drill) rather than to conscious changes to the curriculum made by the health educator (eg, they did not teach a portion of a session because they did not feel the students would respond well).

Experienced HEs were more likely to have their session rated as good, but less likely to perceive the session as good themselves. This result indicates that HE experience is likely to be an important covariate in future analyses designed to compare the effects of the 2 curricula. We expect that classes taught by more experienced HEs will be more effective than those taught by less experienced ones. The process data support this expectation by showing that observers rated sessions taught by experienced HEs better than those taught by less experienced ones. The difference between HE assessment and observer ratings of classroom sessions indicates that experienced HEs are more critical of their performance than are inexperienced ones.

We did not find support for our expectation that there would be more talking about topics other than the curriculum in TND-Network than in TND. HE assessments reported less talking in TND Network than in TND standard, and experienced HEs reported more talking than inexperienced ones did. This result was not replicated in the observer analysis, however. Clearly iatrogenic effects such as class disruptions and off-task talking are a concern and may mitigate program effects expected with increased peer-to-peer interactivity. We did not find evidence for such negative effects in these process data as evidenced by reports of off-task talking. This could be due to the nature of the population receiving the program. These students are already at high risk for substance use and are considered somewhat deviant from mainstream; thus the entire class is “segregated” from society, and the groups were not aggregated problem students separated from nonproblem students because all students are considered problem students.

It is also possible that off-task talking is easier to record in a “standard” classroom setting than in a class divided into groups. When teachers teach to a full class, they expect all attention to be on the teacher and directed to the front of the room. Any talk not addressed to the teacher would be considered off-task and perhaps disruptive. In contrast, a class divided into groups may have many discussions occurring simultaneously, and it would be difficult for a teacher to notice which ones are off-task and which ones are being conducted as part of the exercise. Consequently, the finding that HEs and observers recorded more off-task talking in TND than in TND Network may not indicate that there were more disruptions in TND.

In exploratory analyses, we found a strong positive association between favorable classroom processes and reports of off-task discussions (AOR=9.75, 95% CI=6.94–13.7). Thus, it seems that the creation of a better classroom environment can lead to tangential comments and discussions that are not on task. Experienced HEs may have been more likely to notice this and attempt to get the class back on track than were inexperienced ones. Thus one implication of this study is that it may take experienced HEs to recognize off-task discussion and have the experience to cope with this problem.

Most teachers value classroom control and frown on off-task discussion to maintain classroom discipline and keep it on track. On the other hand, some independence can also be valued as teenagers learn and develop identity through social interaction. Finding the right balance between control and independence is a constant struggle for teachers, parents, and managers. In the context of a continuation school, however, control is usually thought to be beneficial, but of course the rationale for the TND Network curriculum is to turn some of that control over to the students. Our findings, in part, may indicate the difficulty of highlighting the trade-off in benefit between a disciplined, orderly, but perhaps uninterested classroom and one in which some independence is allowed, yet off-task discussion is frequent.

These results reduce our anxiety that peer-led programs in alternative (high-risk) settings will automatically be associated with off-task discussions and uncontrollable classroom environments. Instead, the data indicate that TND was perceived to be easier to implement yet also associated with more off-task discussions. It seems that students will talk among themselves regardless of what is happening in the classroom, and the interactive curriculum provides something for them to talk about whereas a less interactive one creates more opportunity for students to be disruptive.

Finally, there was no difference in quit reports between curricula or by HE experience indicating that, in these process data, there is no evidence for differential effectiveness of the programs.

In sum, these data show differences in implementation between TND and TND-Networked. More favorable classroom processes were reported in TND, and HE experience has emerged as a significant covariate of perceptions of curriculum delivery. There is a trade-off between favorable classroom processes and off-task discussions such that increased student interest and engagement can lead to tangential discussions, joking, and horseplay. Experienced HEs may be needed to control these digressions and keep the class on task to complete the lessons. Whether these processes affect program outcomes will be evident in effects analysis.

Figure 1.

Process Evaluation Study Design

Appendix A

Project TND 2: Health Educator Assessment and Observer Monitoring Form

Regular TND  _________                               Networked TND  _________

School:__________________________________

Name of Health Educator Delivering Session: __________________________________

Period#________________          Session #_________________

Data Entry Person’s Name: ________________________   Date: ______________
__ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ __ 

Implementation:

1. Lesson Taught as Written: How closely did you follow each activity as written?

              Not close            Moderately                 Very      Activity not Activity
               At all                close                    close      in session  omitted

Activity #1      1           2         3            4           5            6         0
Activity #2      1           2         3            4           5            6         0
Activity #3      1           2         3            4           5            6         0

Scale: Taught as Written
1= Not close at all              2= Moderately close                5= Very close
Most of the activity was         Part of the activity was omitted.  No part of the activity was
omitted.                                                            omitted.
The content was presented in     Part of the content was presented  The content was presented in the
a teaching style different from  using a different teaching style   teaching style that was specified
that specified in the guide.     than specified in the guide.       in the curriculum guide.

You presented a substantial      You presented a little bit of con- You did not add any additional
amount of content that was not   tent that was not written in the   content that was not written in the
written in the curriculum guide. curriculum guide. (eg, personal    curriculum guide.
                                 stories, additional examples)
Many key points were omitted.    A few key points were omitted.     All of the key points were made.

2. Were one or more activities shortened?                            Yes No
    2a. If yes, describe:__________________________________________________

3. Were one or more of the activities presented out of sequence?     Yes No
    3a. If yes, describe: ________________________________________________

4. Was the content of any activity presented in a teaching style different from that recommended?
(eg, lecture vs. discussion; classroom discussion vs. small group format?)
                                                                     Yes No
      4a. If yes, describe: ________________________________________________

5. Did you add any activities or content to the lesson?
                                                                     Yes No
      5a. If yes, describe: ________________________________________________

6. Did the lesson seem rushed?
                                                                     Yes No

7. Did the lesson seem to go on too long?
                                                                     Yes No

8. Were there any interruptions that might have affected this lesson (eg, fire drill, shortened time
period, many announcements on PA, etc.)?
                                                                     Yes No
    8a. If yes, describe: _______________________________________________

9. Overall, how well did the students seem to interpret the objectives of the session?

                 1            2          3            4               5

Scale: Interpretation of Objectives by students
1= Not well at all             3= Moderately well                   5= Very well
Few or none of the students    A little over half of the students   All or almost all of the students seemed
seemed to understand the       seemed their inappropriate           to understand the purpose and key
purpose, or the key points     responses) to understand             points of the lesson. Almost all of the
of the lesson. (this might     the purpose and key points           students conveyed this through their
be evident by their lack of    of the lesson.                       appropriate and relevant responses
responses to questions, or                                          to the materials.
their inappropriate responses)
Many students had questions    Some students had questions about    All or almost all of the students who
about the lesson because they  the lesson because they didn’t       had questions seemed to get
didn’t understand the concept, understand a concept, and still      clarification from the health educator,
and still seemed confused even seemed confused even after the       and understand the concept better.
after the health educator      health educator addressed their
addressed their questions.     questions.

Classroom Process:

10. How interested were the students in this lesson?
                 1             2              3         4              5

Scale: Student Interest
1= not interested at all         3= Moderately interested            5= Very interested
None or very few of the          A little more than half of the      All or nearly all of the students seemed
students paid attention to       students paid attention. Some       very interested and listened actively
you, the lesson, and             students were distracted; others    to you, classmates, etc.
classmates, etc.                  appeared to be interested.
All or nearly all students       A little more than half of the      All or nearly all students took the
laughed inappropriately          students took the program           program content seriously.
and/or made silly comments       content seriously.
about the lesson.
All or nearly all of the stu-    A little more than half of the      All or nearly all students participated
dents look bored, disinterested. students participated. Some         actively in the activities and volunteer-
Few participated; few vol-       volunteered to answer questions;    ed responses to your questions.
unteered, and most were re-      some were doing other things
luctant to speak.                such as daydreaming.

11. To what extent did you elicit students’ responses?

                1            2             3          4                5

Scale: Elicit Students’ Responses
1= Not at all                 3= Somewhat                            5= A great deal
You focused on the same       You elicited participation from        You facilitated broad student
students throughout the       more than half of the students in      participation (called on different
lesson; you made no attempts  the class. Sometimes you called        students, called on students who were
to involve students who were  on the same students; sometimes        being less active, made sure everyone
not volunteering comments.    you called on those who had not        participated in group activities).
You allowed students to opt   yet participated.
out of group activities.

12. Overall, how well controlled was the class during the session?

                 1            2           3           4                5

Scale: Classroom Control
1= Very poorly controlled      3= Moderately controlled              5= Very well controlled
Few or no students followed    A little more than half of the        All or nearly all students followed
instructions.                  students followed instructions.       instructions.
All or nearly all students     A little more than half of the        All or nearly all students followed the
talked amongst themselves      students paid attention.              lesson.
in ways that disrupted the
class; are rowdy.
You had to shout to maintain   You had to calm some unruly           You always maintained control; you
discipline and even then the   students down, and activities         didn’t need to shout throughout the
class was still out of control were somewhat disrupted.              lesson; you didn’t have to ask for
(throughout most of lesson).                                         quiet; no activity was disrupted by
                                                                     students.
13. How confident did you feel during the session delivery?
                1            2           3             4               5

Scale: Teacher Confidence
1= Not at all confident       3= Moderately confident                5= Very confident
You felt insecure or unsure   For the most part, you felt con-       You felt very confident. You were
of your abilities all or most fident, but at times you felt          sure of yourself and abilities
of the session.               unsure of yourself or abilities.                      throughout the lesson.

14. How well prepared were you for lesson delivery?
                1           2            3             4                5
Scale: Teacher Preparation
1= Very poor                 3= Moderately prepared                  5= Very well prepared
You were not well prepared   You felt moderately prepared to         You were well prepared (knew lesson
(didn’t know lesson content, teach (were somewhat fluent with        content well; lesson well organized;
lacked the materials needed  the lesson content; lesson pro-         had all necessary materials ready).
for the class).              ceeded in an organized fashion).

15. To what extent did the students challenge the ideas that you presented?
                1            2            3           4                 5
Scale: Challenge Your Ideas (Note: This direction of this scale differs from those above; ie, here a 1 is good
and a 5 is not good.)
1= Not at all                  3= Somewhat                           5= A great deal
The students didn’t challenge  For the most part, students           The students questioned all or most of
any of your comments. They     accepted the key points of the        the key points that you made. They
appeared to accept all of the  lesson without questioning            challenged you in ways that suggest
key points in the lesson. Non- them. Their nonverbal cues            that they didn’t “buy” the key
verbally, they appeared to be- suggested that they believed          points of the lesson.
lieve the comments that you    most of what you and class-
and their classmates made.     mates said.

16. To what extent did you notice students talking about the program?
                 1           2           3            4                5

Scale: Extent to which students discussed session
1= Not at all                   3= Somewhat                          5= Very much
You didn’t hear any students    You heard a few students dis-        You heard most of the students
talking amongst themselves      cussing the session, materials,      discussing the session, materials, or
regarding any part of the pro-  or program sometime during           program sometime during the class
gram, session, or materials at  the class period.                    period.
sometime during the class
period.
No students discussed the       A few of the students discussed      Most of the students discussed the
activities together in order to the activities together in order to  activities with other individuals to
come up with responses. If come up with responses. You               come up with responses. You heard
any responses were given to     heard a few side comments from       many side comments from several
you, the individuals came up    a few students regarding the         students regarding the session,
with the ideas on their own     session, materials, or program.      materials, or program at some point
with no verbal discussion.                                           during the class period.

17. To what extent did you notice students talking in their groups about things other than the
program?
                1            2            3            4               5

Scale: Extent to which students talked about other things (Note: This direction of this scale differs from
those above; ie, here a 1 is good and a 5 is not good.)
1= Not at all                 3= Somewhat                            5= Very much
The only talking that you     There was little, but some talking     There was extensive talking between
heard go on between students  between students about things          students about things unrelated to the
was about aspects of the pro- other than the program, session,       program, session, or materials.
gram, sessions, or materials. or materials.

18. During the classroom session, did any student state a desire to not begin use, cut down on use, or
quit use of drugs?
     Yes       No
     
     18a. If yes, about how may? _________

19. What is your OVERALL/GLOBAL rating of how this session went?
    Good (5)     Average (3) Poor (1)

{If you have any other comments about the session, please write them on the back of the form.}