Adolescents’ impressions of antismoking media literacy education: qualitative results from a randomized controlled trial

Abstract

Although media literacy represents an innovative venue for school-based antismoking programming, studies have not systematically compared student impressions of these and traditional programs. This study utilized data from a randomized trial comparing these two types of programs. After each program, students responded to three open-ended questions related to their assigned curriculum. Two coders, blinded to student assignments, independently coded these data. Coders had strong inter-rater agreement (kappa = 0.77). Our primary measures were spontaneously noted overall assessment, enjoyment/interest and the likelihood of changing smoking behavior. Of the 531 participants, 255 (48.0%) were randomized to the intervention (media literacy) group. Intervention participants had more net positive responses [rate ratio (RR) = 1.27, 95% confidence interval (CI) = 1.05, 1.54], more responses rating the program as compelling (RR = 1.63, 95% CI = 1.16, 2.29) and fewer responses rating the program as non-compelling (RR = 0.62, 95% CI = 0.39, 0.97). However, the intervention group was not more likely to suggest that the curriculum was likely to change behavior positively (RR = 0.57, 95% CI = 0.30, 1.06). Findings suggest that although media literacy provides a compelling format for the delivery of antitobacco programming, integration of components of traditional programming may help media literacy programs achieve maximal efficacy.

Introduction

Smoking is the leading cause of morbidity and mortality in the United States, killing 442 000 people each year [1]. Since the vast majority of those who will die from tobacco begin smoking at age 18 or younger [2], many counter-tobacco interventions appropriately focus on youth [3]. An excellent way to reach youth is in a school-based setting since the vast majority of youth attend school [4]. Historically, traditional school-based programs have focused on teaching the short- and long-term effects of smoking and helping young people develop skills to avoid tobacco use through counter-argument [3, 5–8]. These programs, however, are not generally compelling to youth and have failed to result in long-term reduction in tobacco use [3, 5–9]. As an example, one particularly large well-funded longitudinal school-based curriculum of 47.25 hours administered >8 years failed to result in decreased smoking rates [8]. In that study, smoking rates were no different in the experimental and control conditions among boys, girls, high-risk individuals or low-risk individuals [8].

Since the development of these programs, however, the strategy of ‘media literacy’ as an antitobacco tool has been recommended by organizations such as the American Academy of Pediatrics, the Centers for Disease Control and Prevention and the Office of National Drug Control Policy [2, 10, 11]. These recommendations have been borne out of the discovery that media exposure to smoking contributes strongly to initiation of adolescent smoking, whether that exposure occurs in narrative contexts such as films [12–14] or in persuasive contexts such as advertising and promotion [15–18]. Although one method of reducing adolescent smoking would be to reduce exposure to media representation of smoking [12, 19, 20], it is not always possible and/or feasible to do this [21–25].

Thus, media literacy, often defined as the ability to understand, analyze, evaluate and create media messages in a wide variety of forms [26–28], represents a potentially powerful behavior change strategy available to public health advocates. Theoretically, media literacy may be more accessible and more effective than traditional programs. First, media literacy may be more acceptable to adolescents—who often find traditional programs to be monotonous, old-fashioned or ‘preachy’ [9]—since it involves the use of media examples that may be compelling, interesting and relevant to young people. Once youth are ‘hooked’ by this more accessible format, they may be less likely to ‘tune out’ from the antitobacco message. Second, media literacy may be effective among adolescents because of its ability to alter smoking attitudes and smoking normative beliefs, constructs that have been closely linked to smoking behavior both by the Theory of Reasoned Action [29] and by empirical data [30–33]. Media literacy may be able to do this because it sensitizes young people to the manipulative and often implicit messages designed to confer pro-smoking attitudes and subjective norms. Thus, using media literacy to examine the careful construction of pro-tobacco messages and salient omissions may lead to decreased belief in the validity of these implied effects of smoking [29].

A recent comprehensive review of media literacy interventions shows that it has been used for a variety of purposes [34]. Although media literacy has been used most frequently as an intervention related to eating behaviors [35–40], media literacy programs have also focused on violence, nutrition [36, 41] and substance use [42–44]. Early antismoking programs were well liked by students but have not had evaluations in the peer-reviewed literature [11, 45]. Since that time, others have evaluated antismoking media literacy programs and found that media literacy programs can help young people increase reflective thinking regarding tobacco-related media messages [43, 44]. These programs also have been successful at altering variables related to tobacco use such as normative beliefs related to smoking [43, 44]. Additionally, studies measuring media literacy in adolescents have found it to be strongly and independently associated with both reduced adolescent smoking and reduced susceptibility to future smoking [46, 47]. However, studies have yet to rigorously and directly compare students’ impressions of media literacy programs versus traditional harm-based programs.

The purpose of this study was to compare qualitatively analyzed student impressions of two different types of antismoking education: media literacy and traditional programs. Based on the theoretical arguments described above, we hypothesized that the media literacy program would be more compelling and acceptable to young people, as evidenced by more net positive overall responses (Hypothesis 1). We also expected more descriptions of media literacy as fun/interesting (Hypothesis 2) and more likely to reduce smoking behavior (Hypothesis 3). We also expected more responses from media literacy participants exhibiting critical thinking (Hypothesis 4). We expected no significant difference with regard to codes related to information (Hypothesis 5), format (Hypothesis 6) and logistics (Hypothesis 7) since we endeavored to match the curricula on these characteristics.

Methods

Study context

This study utilized data from Year 1 of a randomized controlled trial investigating the efficacy of media literacy education. Over the 2006–07 school year, all 9th grade students from two large urban public high schools were eligible for enrollment. These students represented 28 health classrooms, each of which was randomized to either the intervention (media literacy education) or control (traditional) condition. Randomization was stratified by time of day and regular classroom teacher and was conducted in blocks of four classrooms each. On all evaluation instruments, students used unique non-traceable identification numbers to keep their identities confidential. This study was approved by the University Institutional Review Board (IRB #0606146).

Curricula

Each curriculum consisted of three weekly sessions consisting of one class period (∼45 min) each. The experimental condition was the three-session version of the ‘AD IT UP’ curriculum—a theory-based antismoking media literacy curriculum developed and pilot tested with foundation grant funding. This curriculum teaches youth to access, analyze and evaluate mass media messages involving smoking, such as advertisements, promotions or film product placements. Each letter of the title refers to one of the six lessons contained in the curriculum, each of which is associated with a ‘core concept’ based upon a composite theoretical model of media literacy and a media literacy ‘key question’ (Table I).

Table I.

Relationship between the ‘AD IT UP’ curriculum and the theoretical model of media literacy

Media Literacy Domain	Related Media Literacy Core Concepts	AD IT UP key question
AA: authors and audiences	AA1: authors create media messages for profit and/or influence	Who is the Author of this media message, and what is their motive/purpose?
	AA2: authors target specific audiences	Who is this message Directed against? (That is, who is the target audience?)
MM: messages and meanings	MM1: messages contain values and specific points of view	What Ideas are they trying to get across through this message?
	MM2: different people interpret messages differently
	MM3: messages affect attitudes and behaviors	What Techniques do they use to make this message in order to get those ideas across?
	MM4: multiple production techniques are used
RR: representation and reality	RR1: messages filter reality	What is Unspoken or omitted from this message?
	RR2: messages omit information
	Facilitation of movement from altered attitude and norms toward intention and behavior	What is your Plan now that you know what you know? Is this something you want to buy/do or not?

This composite theoretical model combines constructs from two paradigms [28, 48]. Although they overlap substantially, there are differences in emphasis. One model emphasizes that (i) the purposes of media producers and characteristics of target audiences should be carefully considered, (ii) there are multiple complex production techniques and symbol systems used to convey meaning and (iii) there is a complex relationship between media representations and social reality [48]. Another model emphasizes that (i) media messages are carefully constructed with the use of their own complex language, (ii) different individuals interpret messages differently, (iii) messages contain inherent values and perspectives and (iv) media messages are usually created for profit and/or power [28]. Integrating both of these models into a comprehensive theoretical framework fully captures the construct of media literacy [47].

The control curriculum was based on the three-session antismoking program featured in the most widely used and commonly accepted health textbook in the region of the study [49]. These three lessons, which form the quintessential traditional program, describe both short- and long-term effects of smoking, support students’ self-efficacy in saying ‘no’ to smoking and include a discussion of social and psychological issues involving smoking. Thus, the programs differ in that the media literacy program is more focused on developing critical thinking skills around tobacco-related media messages, whereas the control curriculum focuses on the harm of smoking and skills to avoid smoking.

The curricula were implemented by experienced health educators who were not the students’ regular high school teachers. The instructors were given appropriate training, consisting of a total of four 3-hour sessions over the course of 3 months. The first hour of each session consisted of didactic information related to the curricula and demonstrations of the curricula. Then, during the latter 2 hours, the instructors were given opportunities to practice conducting the curricula and to receive feedback from their fellow instructors. Finally, each instructor had an opportunity to practice implementation on convenience samples of high school students from other (non-study) schools before actual implementation. Each teacher remained with the same cadre of students for each of the three lessons in order to maintain teacher–student rapport and continuity.

Because of the intrinsic differences between media literacy and traditional antismoking education, the information presented was different. However, the intervention and control conditions were matched in as many other ways as possible. Each of the teachers taught both curricula. The students were given the same pre- and post-intervention surveys. The curricula featured the same format—Power Point presentation combined with activities—featuring similar numbers of slides and numbers of images in each presentation. Workbooks were also matched: both curricula featured a 16-page workbook with similar activities (although with different content). Workbooks featured activities such as open-ended thought questions, matching exercises and fill-in exercises based on the material presented in class.

Instructors self-evaluated their implementation fidelity after each teaching session. Composite implementation fidelity scores showed 98.7% fidelity to the intervention curriculum and 99.1% fidelity to the control curriculum. Thus, fidelity did not seem to be skewed by curriculum type.

Data collection procedures

All participants completed a post-test 1 week after completion of the 3-week curricula. On these post-tests, students responded to three open-ended questions assessing these curricula. In order to get an unbiased assessment, we avoided asking leading questions such as ‘Do you think this program would make kids not smoke?’ Instead, the three items for which we analyzed responses included the following: (i) ‘What did you like about this program?’, (ii) ‘What did you not like about this program, or what did you think could be improved?’ and (iii) ‘What other comments do you have about this program?’ Students typed their responses directly on to computers so they could not be identified by handwriting, and they were not required to respond for credit. We used these methods to maximize the validity of our data. Although participants also provided quantitative data assessing tobacco-related constructs that will be used for later longitudinal analyses, these data were not relevant to the current study, which aimed instead to capture students’ impressions of the programs in their own words. Overall, the post-test was completed in an average of 20 min.

Coding process

We employed a quasi-statistical coding mechanism [50]. Such ‘mixed methods’ approaches can be valuable in bringing statistical rigor to data obtained via qualitative coding [51–54]. We first developed a tentative list of codes based on our research questions and hypotheses and on prior studies, qualitatively assessing smoking-related issues in adolescents and young adults [55–60]. Four study personnel, blinded to participant group assignment (intervention or control), then individually coded the first 10% of responses using this codebook. Based on these initial reviews and on a discussion process between the four coders, we developed a final study codebook, condensing similar or redundant codes and using a system of open codes that incorporated new themes as they emerged from the text.

Two coders then independently coded data from all 531 participants using Atlas.ti, a qualitative software analysis program [61]. Coders were blinded to participant group assignment throughout the coding process. When both coders were finished, we assessed the inter-rater reliability of our coding using Cohen’s kappa scores. The average kappa score was 0.77 which Landis and Koch [62] describe as a substantial level of inter-rater reliability. Finally, every discrepancy between the two coders was discussed among the team until each discrepancy was adjudicated, resulting in a final data set. Only after the list of codes was finalized were coders unblinded.

Coding scheme

Our final codebook contained 34 relevant codes (Table II). As part of our coding scheme, we included both a single overall assessment which was applied to all cases, as well as a thematic analysis where codes were applied as topics were expressed.

Table II.

Code and quotation examples

Code	Quotesa	Control/intervention
ACTIVITY BAD	[I didn't like] when we had to guess the percentage of things. I wouldn't have some people guess i woulg just get facts	Intervention
ACTIVITY GOOD	I liked the parts where we had to figure out the subliminal messages.	Intervention
BORING	it needs to be more fun. kids won't listen if their bored.	Control
CHANGE GOOD	it might help people stop smoking	Control
	The speaker was really nice. I learned a lot from this program. I am less likely to smoke now.	Control
	I am a athlete and I am never ever ever ever ever ever ever ever going to smoke	Control
	I believe the presentations and program was well thought out and that it would be successful to get kids to stop smoking or to never start smoking.	Control
	it was a big help to me and my class because now that i see the real smoking problems, im more likely to stay away from tabacco	Control
	they should do this progam for adults cause maybe it would make a few people quit	Control
	That I found out how many diseases people may recieve because of cigarettes. I then thought how many of my friends smoke cigarettes, and how much of them they do. It encouraged me to get them to stop.	Control
	This program encoraged me not to smoke in the future.	Control
	What I enojoyed most was learning more about the harm of cigarettes and what they can do to you because that makes me not want to smoke even more.	Control
	I think it may help teenagers stop smoking them to smoke.	Intervention
	Thankyou I now can pass on this information to my parents to get them to quit smoking.	Intervention
	i enjoyed how they asked me if i smoked and i now will never smoke.	Intervention
	they show keep this program going cause it could save alot of kids.	Intervention
	Lerning. I learned about the different diseases and physical and mental injuries from smoking which will keep me off cigarettes for the rest of my life.	Intervention
	This program was truely helpful and even though I hadn't intended on smoking in the future anyway, I now have a more motivative reason not to smoke because of this program.	Intervention
	Looking beyond the ads and seeing them for what they really are. Its good to know because now its gonna be harder for to get sucked into those stupid advertisments.	Intervention
	it was areally nice program and i think that it can really make a big difference n people my age's lives.	Intervention
CRITICAL THINKING	that it showed us to look behind the smiling faces in advertisments, and to look deeper and how the product would really affect you.	Intervention
FUN	It was fun!	Control
	FUN FUN FUN!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!	Intervention
INFO BAD	needs better topics	Control
	[I didn't like] the information	Intervention
INFO HEALTH	it let me know the serioius side effects of smoking	Control
	I enjoyed lerning more about the health efects of somking	Control
	I liked how the program informed me about what I'm doing to my body and my overall health.	Intervention
INFO OTHER	I enjoyed the information provided for us by the speaker.	Control
	I enjoyed learning about the hidden messages in the advertisements.	Intervention
INTERACTION GOOD	i liked how u can talk about your fellings	Control
	[I liked] The involvement of the students during the program	Control
	i enjoyed talking about different things	Intervention
INTERACTION NOT ENOUGH	I didnt like how it was so long and that we couldnt see how it affects you we just heard about how it affects you.	Control
	What I enjoyed least was not getting to interact as much. We answered a lot of questions but I wanted to do more.	Control
	It was kinda boring because it was a little to much talking. If we would of had more activites to do then it would have been easier for me to keep focus.	Intervention
INTERESTING	UMM … iT WAS A REALLY GOOD ATTENTIONS GRABBER	Control
	it was intereting	Intervention
LESS	going threw it a little faster might help	Control
	Sometimes it took to long to get to the next subject	Intervention
MORE	I leat enjoy the fact it's over so fast.	Control
	the way it could be improved is by making the program longer than just 3 sessions because this program really kept my attention.	Intervention
NEGATIVE	he just told me everything i already knew	Control
	I didnt enjoy anything about this program	Control
	nothing i hate smoking programs they annoy me they dont keep my attention	Control
	I didn't enjoy all the talking because I had already heard most of the stuff.	Intervention
	it did not grab my attention like i wanted it to.	Intervention
NEUTRAL	IDK	Control
	??????????????????????	Control
	i have no comments	Intervention
NO NEW INFO	he just told me everything i already knew	Control
	[I didn't like] The things that I already knew being repeated.	Intervention
POSITIVE	I thought it was good that we had this program in our school. Nohting in my mind could be improved I thought it was ok.	Control
	It was a good learing experience for those who did not no much about it.	Control
	it was a great program and i would recommend it to another one of my friends	Control
	I enjoyed learning more about cigaretts and drugs and what else they could do to you.	Control
	[I liked] when we got to express our feelings	Control
	I liked it very much and they should definately come back.	Intervention
	I think this program is very informative and it shows young, impressionable children the reasons why you should not smoke.	Intervention
	i think it was fun and everything i enjoyed it. it was a good learning program of smoking, and doing what u are not suppose to do.	Intervention
	I felt that the program didnt really need anything for it to be improved.	Intervention
SURVEY BAD	improved could be this boreing servey	Control
	this survey is really boring	Intervention
SURVEY GOOD	Taking the Survay. I think it's fun just to get your idea out to people or whoever will review this.	Control
	i enjoyed how we got to guess an the sureys	Intervention
TEACHER BAD	He [the teacher] spoke in the same tone the whole time he was speaking. More enthusiasm, or engery.	Control
	I think that you need a more interesting presenter, he didn't seem interested.	Intervention
TEACHER GOOD	The teacher was nice, and made everything he taught easy to understand :]	Control
	i like our teacher that took the time and tell us about these left out facts	Intervention
VISUAL BAD	[I didn't like] the pictures of the lungs and other pictures.	Intervention
VISUAL GOOD	I like the pictures and ads and how we noticed what and who they were trying to single out.	Intervention

^aQuotations are reproduced verbatim, including any typographical errors.

Primary codes (Hypotheses 1–3): overall assessment, enjoyment and interest and likelihood of changing smoking behavior

As our first primary coding set, coders assigned an ordinal overall assessment code (Hypothesis 1) of ‘POSITIVE,’ ‘NEUTRAL’ or ‘NEGATIVE’ to each of the three participants’ responses (corresponding to Questions 1, 2 and 3, respectively). This method was chosen as students frequently included negative comments in the ‘most liked’ category or positive or neutral comments in the ‘least liked’ category, with the ‘other’ category having various observations, either positive, negative or neutral. Each of the three response categories completed by the students were judged to be either positive, negative or neutral (Table II). Kappa values for each of these codes were ‘nearly perfect’ [62] at 0.88, 0.79 and 0.89, respectively. We converted this qualitative code set into a quantitative value by subtracting any negative scores from the positive scores. (Neutral codes were not included since they were given a value of zero.)

For the second primary coding set, we focused on the interest level of the course (Hypothesis 2). For this and all subsequent coding, we used a thematic analysis approach, applying codes as textual themes emerged. Here, codes related to enjoyment of and interest in the curricula were labeled ‘FUN’ and ‘INTERESTING’, and, conversely, the code labeled ‘BORING’ related to a lack of enjoyment or boredom (Table II). Coders marked text with the code labeled ‘MORE’ when participants expressed a desire for additional or longer sessions, and they used the code ‘LESS’ when the text indicated that the program was too long or contained too many sessions. A composite ‘compelling’ coding score summed instances of FUN, INTERESTING and MORE, whereas a composite ‘non-compelling’ coding score summed BORING and LESS.

Our third coding set focused on whether the curricula promoted changed behavior (Hypothesis 3). We used the code ‘CHANGE GOOD’ to mark statements that spontaneously suggested that the curriculum would be likely to change behavior in a positive way among the respondents or their peers. Although we also searched for instances of ‘CHANGE BAD’ which would indicate that the curricula might make them or their peers more likely to smoke, we did not find any instances of this code.

Secondary codes (Hypotheses 4–7): critical thinking, curriculum content, curriculum format and logistical aspects

‘CRITICAL THINKING’ was a single code used to indicate a process of critical analysis (Hypothesis 4). This code was used if, for instance, participants used language such as ‘figured out’ or ‘learned how to look at’ a given issue in the curriculum (Table II).

Another series of secondary codes related to curriculum content (Hypothesis 5). These included ‘INFO HEALTH’, ‘INFO OTHER’, ‘INFO BAD’ and ‘NO NEW INFO’ (Table II). A total content composite score was created summing INFO HEALTH and INFO OTHER and subtracting NO NEW INFO and INFO BAD.

Codes related to the format of the curricula (Hypothesis 6) were related to activities, interaction and visual elements. ‘ACTIVITY GOOD’ and ‘ACTIVITY BAD’ were used only when the format of a specific activity was mentioned. Three codes related to interactivity during the program: ‘INTERACTION GOOD’ indicated that a participant liked the degree of interactivity, whereas ‘INTERACTION NOT ENOUGH’ indicated that it lacked sufficient interactivity. Finally, ‘VISUAL GOOD’ and ‘VISUAL BAD’ indicated whether they appreciated or disliked visual elements of the curricula.

Finally, several codes related to logistical aspects (Hypothesis 7) of the programs. For instance, ‘TEACHER GOOD’ and ‘TEACHER BAD’ indicated positive and negative comments about the program instructor and ‘WRITING GOOD’ and ‘WRITING BAD’ commented upon the workbooks that went along with each of the curricula. We coded ‘BLANK’ when a section was blank or otherwise not useful.

Analysis

The frequencies of the codes were assessed using a quasi-statistical qualitative methodology [50]. We summed the number of counts for each code among those in the intervention and the control conditions. We then computed rates by dividing each code count by the total number of participants in each study arm. Finally, we used Stata Statistical Software [63] to compute the rate ratio (RR) for each code, comparing intervention with control, and its 95% confidence interval (CI) since we considered statistical significance as P < 0.05.

Results

Sample

Our sample consisted of the 531 students who completed the post-test. The sample was 50.5% female with a mean age of 14.9 (SD = 0.9). Of the 532 participants, 255 (48.0%) represented the 14 classrooms randomized to the intervention group and 276 (52.0%) represented the 14 classrooms randomized to the control group. Those in the intervention group were no more likely to be male than female (P = 0.09), to be non-White (P = 0.54) or to be older (P = 0.40). There were 1593 total fields to be coded (three for each individual corresponding to the three open-ended questions). Because there were a total of 242 fields without valuable data, the overall response rate was 84.8%.

Primary codes (Hypotheses 1–3)

Overall, intervention participants had more net positive responses [RR = 1.27, 95% CI = 1.05, 1.54]. Intervention participants also had more responses rating the program as compelling (RR = 1.95, 95% CI = 1.23, 3.13) (Table III). Similarly, intervention participants had fewer responses rating the program as non-compelling (RR = 0.62, 95% CI = 0.39, 0.97). However, the intervention group was not more likely to spontaneously suggest that the curriculum was likely to change behavior positively. In fact, the data exhibited a non-significant trend toward intervention participants being less likely to suggest positive change impact of the curriculum (RR = 0.57, 95% CI = 0.30, 1.06).

Table III.

Code counts and rate ratios

Code	Definition	Intervention counts (n = 255 participants)	Control counts (n = 276 participants)	Rate ratio (95% CI)a
1. Overall assessment
POSITIVE	Any response with an overall positive connotation	368	375	1.06 (0.91, 1.22)
NEUTRAL	Any response with an overall neutral connotation	220	224	1.06 (0.88, 1.28)
NEGATIVE	Any response with an overall negative connotation	128	172	0.80 (0.63, 1.01)
TOTAL OVERALLb	POSITIVE − NEGATIVE	240	203	1.27 (1.05, 1.54)*
2. Enjoyment and interest
FUN	Thought the program was fun or enjoyable	44	26	1.82 (1.10, 3.09)*
INTERESTING	Thought the program was interesting	12	5	2.59 (0.85, 9.38)
MORE	Wanted more sessions, longer sessions or would want to attend again	36	30	1.29 (0.77, 2.17)
TOTAL COMPELLINGb	FUN + INTERESTING + MORE	92	61	1.63 (1.16, 2.29)*
BORING	Thought the program was boring or not fun	24	49	0.53 (0.31, 0.88)*
LESS	Thought the program was too long or that there were too many sessions	9	8	1.21 (0.42, 3.61)
TOTAL NON-COMPELLINGb	BORING + LESS	33	57	0.62 (0.39, 0.97)*
3. Efficacy
CHANGE GOOD	Indicated that smoking behavior of self or others would be likely to change as result of having participated in the program	17	32	0.57 (0.30, 1.06)
4. Critical thinking
CRITICAL THINKING	Used language showing critical analytic or thinking skills	30	2	16.17 (4.10, 139.66)*
5. Content
INFO HEALTH	Learned information specifically pertaining to health issues	37	68	0.59 (0.38, 0.89)*
INFO OTHER	Learned information not pertaining to health	72	67	1.16 (0.82, 1.64)
NO NEW INFO	Already knew a lot of information presented and/or did not learn much new information	3	10	0.32 (0.06, 1.26)
INFO BAD	Did not like the information presented	5	6	0.90 (0.22, 3.53)
TOTAL CONTENTb	INFO HEALTH + INFO OTHER − NO NEW INFO − INFO BAD	101	119	0.92 (0.69, 1.20)
6. Format
ACTIVITY GOOD	Recalled and enjoyed a specific activity	25	1	26.95 (4.41, 1106.64)*
ACTIVITY BAD	Did not like a specific activity	2	0	N/Ac
TOTAL ACTIVITYb	ACTIVITY GOOD − ACTIVITY BAD	23	1	24.80 (4.03, 1021.47)*
INTERACTION GOOD	Liked interactive elements of program (hands-on activity, interactive discussion)	9	17	0.57 (0.22, 1.35)
INTERACTION NOT ENOUGH	Indicated that the program should have more interaction	11	18	0.66 (0.28, 1.47)
INTERACTION BAD	Did not like a interactive elements of program (hands-on activity, interactive discussion)	18	14	1.39 (0.65, 3.01)
VISUAL BAD	Disliked specific visuals of the program or wanted less of them	15	20	0.81 (0.39, 1.66)
VISUAL GOOD	Liked specific visuals or indicated wanting more visuals	40	21	2.05 (1.18, 3.67)*
7. Logistical details
TEACHER GOOD	Used positive language in describing the teacher, such as teacher was nice or knowledgeable	14	38	0.40 (0.20, 0.75)*
TEACHER BAD	Used negative language in describing the teacher, such as teacher was boring or unable to keep attention	7	8	0.94 (0.29, 2.98)
TOTAL TEACHERb	TEACHER GOOD − TEACHER BAD	7	30	0.25 (0.09, 0.58)*
SURVEY GOOD	Liked the survey instrument	3	4	0.81 (0.12, 4.78)
SURVEY BAD	Did not like the survey instrument	8	3	2.88 (0.69, 16.82)
BLANK	Responded with a blank, an ‘I don't know’ response, or its equivalent	123	119	1.11 (0.86, 1.45)

^aComparing intervention versus control. Raw RRs are computed by dividing (intervention counts/255) by (control counts/276). In this way, these RRs account for the fact that there were different numbers of individuals in the intervention and control groups; 95% CIs were computed using Stata 9.0 (StataCorp, College Station, TX, USA).

^bComposite measure. *P < 0.05.

^cIt is not possible to compute a rate ratio and 95% confidence interval when the control counts are 0.

Secondary codes (Hypotheses 4–7)

Critical thinking was more commonly found among intervention participants (RR = 16.17, 95% CI = 4.10, 139.66) (Table III). Our composite information score was no different in the intervention and control conditions (RR = 0.92, 95% CI = 0.69, 1.20). However, information specifically pertaining to health (INFO HEALTH) was less common among intervention participants (RR = 0.59, 95% CI = 0.38, 0.89). Intervention participants were more likely to mention specific activities in a positive way (RR = 26.95, CI = 4.41, 1106.64). Of the logistical codes, only TEACHER GOOD was different between the two groups; it was less likely to be mentioned by the intervention group (RR = 0.40, 95% CI = 0.20, 0.75).

Discussion

This qualitative assessment of data from a randomized controlled trial found that students considered the media literacy intervention a more positive and compelling experience overall. However, intervention participants were no more likely to state that the intervention might change health behavior; in fact, control group participants exhibited a non-significant trend toward more such statements. While the intervention group seemed to exhibit more critical thinking and more frequently mentioned specific activities they enjoyed, the control group mentioned more health-related information they had learned.

Our finding that media literacy was compelling to students is consistent with prior research [27, 64–66]. One reason for the compelling nature of media literacy programming in adolescence may be that these programs utilize media from popular culture that are highly familiar among and relevant to students. Because they are based in critical thinking pedagogy, media literacy programs also may be more respectful of students, allowing them to make up their own minds instead of being directly told what to do by an authority figure. This ‘low-pressure’ environment may lead to a more relaxed and enjoyable learning experience.

It is interesting that intervention participants were no more likely than control participants to suggest that they or their peers might be less likely to smoke after the intervention. This may be because the control curriculum contained strong attitudinal antismoking messages, some of which can be effective [22, 67]. However, control program participants may have included more responses related to our CHANGE GOOD code simply because they were parroting the unambiguous, harm-related messages of that curriculum.

Similarly, since intervention participants experienced a curriculum emphasizing critical thinking, it is not surprising that those in the intervention group exhibited more language related to critical thinking. This is encouraging since critical thinking skills may provide longer term protection against the effects of ever-changing marketing strategies. Further studies are needed to elucidate the relationship between critical thinking as taught through media literacy education (i.e. critical thinking about media messages) and longer term protection against the ever-changing media marketing strategies for tobacco and other substances.

INFO HEALTH was more frequently coded in the control group than the intervention group. This was expected since the traditional program emphasized specific health-related information, such as how many people each year die from smoking or the names of specific chemicals in cigarette smoke.

Contrary to our hypothesis, TEACHER GOOD was more commonly coded in the control condition than the intervention condition, despite the fact that (i) the intervention had higher overall POSITIVE responses and (ii) the same teacher taught both curricula. This may have been because control group participants had fewer other memorable statements to make, so they praised the instructor instead. The fact that TEACHER BAD was equally common in both groups supports this conclusion. Another possibility may be that the teacher appeared more knowledgeable in the control condition because he was able to give unambiguous facts and figures related to smoking. TEACHER GOOD quotations such as ‘the instructor knew a lot of information and was able to answer any questions’ and ‘he was the right person to educate some one [on] this topic … [and] he knows his stuff’ lend credence to this supposition.

Limitations

One limitation of any qualitative methodology is that interpretation and bias introduced by the researcher poses a threat to internal consistency [68]. We sought to minimize this by using a team of two researchers to independently code the data and comparing their responses with reliability scores. Additionally, although the interventions were delivered at the classroom level, these analyses do not take into account nesting within classrooms. Related to this, it should be noted that since different classrooms were randomized within the same school, there was the potential for contamination (students discussing each others’ curricula after school). Another necessary limitation of our methodology was that although we blinded coders to participant assignment group, it is possible that the coders could guess intervention assignment.

Another limitation of the study was that by quantifying our results and focusing on RRs, we may have lost some of the inherent richness of the qualitative responses. However, such a methodology is valuable, when combined with rich qualitative data (Table II), to lend rigor to our conclusions. Our use of open-ended survey questions, as opposed to more commonly used focus groups or interviews [55, 58, 69–72], may have also been a limitation. This is because, without guidance from an interviewer, adolescents’ open-ended responses could generate irrelevant, ambiguous or sparse information. However, we chose this unique methodology because it allowed us to avoid the bias inherent in leading participants to certain responses. In fact, it represents an innovative method of conducting qualitative research that may be useful to future investigators.

Conclusions

This study is unique in that it qualitatively analyzed open-ended student responses for overall assessment as well as specific content related to the study aims. Although we found the media literacy intervention did seem to be overall a more positive and compelling experience for students, we also found that intervention participants were no more likely to state that the program might be effective. Thus, there seem to be risks and benefits to each type of antismoking education. Although media literacy seems to provide a compelling framework for antismoking education, integration of components of traditional programming into these programs may make them maximally effective.

Funding

National Cancer Institute (K07-CA114315 to B.A.P.); Robert Wood Johnson Foundation (57407 to B.A.P.); Maurice Falk Foundation (Major Grant Award to B.A.P.); Veterans Administration Health Services (Research Development Merit Review Award to S.Z.).

Conflict of interest statement

None declared.