Comparison of Research Designs for Two Controlled Trials of Mass Media Interventions

Abstract

This paper compares two controlled trials of mass media interventions, factors influencing their designs, and design lessons learned from these experiences. Mass media evaluations based on a scientific research model are motivated by gaps in knowledge. The results of such research are intended to serve the needs of consensus development processes through which confident recommendations can be made for intervention strategies that should be more widely applied. For these purposes, the scientific research context emphasizes internal validity of evaluation design, such as controlled experiments. This paper describes two such trials, implemented at different times with differing social contexts for youth cigarette smoking, smoking prevention research evidence bases, and tobacco control environments. Common and unique features of the two trials are reviewed, and observations are noted about the conditions under which controlled trials of mass media interventions might be warranted.

This paper compares two controlled trials of mass media interventions, factors influencing their designs, and lessons from these experiences. A dominant influence on design of these studies was the research context in which they originated. Designs of the studies described here were primarily the product of the investigators’ analyses of gaps in knowledge and research priorities. The joint purpose of the investigators and funding agencies was to develop new knowledge demonstrating as clearly as possible the effectiveness of mass media interventions in reducing youth smoking. New knowledge created through such scientific research studies can be used as a foundation for public policy recommendations concerning design of larger scale public health campaigns or programs.

In this context the investigators were in control of both the media campaign and evaluation designs, and had the capacity to integrate these project components. This approach differs from a situation in which a media campaign is implemented by a government agency, for example, as a component of a large-scale public health program; in that case evaluation often is carried out by a group primarily dedicated to that mission as a separate task.

In the scientific research situation the major motivation is to create strong evidence; in the campaign evaluation situation the major motivation is to demonstrate that the campaign is achieving its objectives. These different perspectives might also be characterized as an emphasis on strength of evidence (internal validity) for research studies and on effects in representative populations (external validity) for campaign evaluations.

The fraction of resources dedicated to evaluation in these two situations differs substantially. In the scientific research situation a common rule of thumb is that up to half of the overall resources will be focused on evaluation and the remainder on intervention. Scientific research studies testing interventions must carefully balance the resources dedicated to interventions (sufficiently powerful to have a reasonable chance of achieving the desired effects) and evaluation designs (sufficiently powerful to have a reasonable chance of credibly demonstrating effects). A cost of balancing these competing priorities is trial interventions that are typically smaller than public health campaigns due to the need to optimize internal validity with complex evaluation designs. In a typical campaign evaluation situation, by comparison, the budget dedicated to the evaluation component of the overall effort would likely be a much smaller fraction, but the population covered by the campaign would likely be much larger and more diverse.

Public policy recommendations based on scientific evidence are heavily influenced by assessments of the strength of evidence supporting a recommendation. Evidence reviews and consensus statements often use highly structured methods to assess study designs, as well as results, in weighing their impact on conclusions and recommendations (e.g., Task Force on Community Preventive Services, 2001). The randomized experimental design, as adapted for clinical trials and similar studies, represents a highly valued standard for assessment of the importance of research study results. A particularly stringent review process might, for example, eliminate from consideration any studies that do not include a randomized control group. In the absence of better evidence, however, other reviews might consider designs that are controlled but not randomized.

Where experimental studies might not be feasible or ethical, alternative designs and other types of evidence can be considered convincing. For example, the universally accepted conclusion that cigarette smoking causes harm is based on massive epidemiologic evidence and animal studies but not on human experiments. Similarly, multiple positive evaluations of interventions with similar objectives and designs may provide convincing evidence of effectiveness despite lack of studies with experimental designs.

By definition a mass media campaign directs standard messages to large populations simultaneously. A simple experimental design requires allocation of individual subjects to different treatment conditions. Reconciling the facts of mass exposure of groups with the requirements for controlled designs required some development of perspectives on criteria for adequate designs. The design implications of applying interventions to entire social groups, such as school or media markets, was well-recognized at the time the first study described here was designed; but in practice this issue was ignored for many community health promotion research studies of that era. The risk of estimating inappropriately small variances eventually was acknowledged, and a more rigorous standard was developed that considered the community, rather than the individual, as the unit of allocation to treatment (Murray, 1998). This evolution had an impact on consideration of experimental designs to evaluate media interventions because of the high costs of conducting such studies with the large numbers of media markets and individual participants required by these designs.

The impact of these considerations will be seen in the following descriptions of two controlled trials of mass media interventions to reduce youth smoking. Several themes recur. The investigators attempted to optimize both the power of the interventions to achieve both demonstrable impact and the quality of the evaluation designs. The intervention and evaluation designs chosen also were influenced by the social context for youth cigarette smoking and prevention at the time, the state of knowledge concerning effective interventions, and the limitations of funding mechanisms.

A Controlled Trial of Mass Media Impact, 1985–1989

The first controlled trial of mass media interventions to reduce youth smoking conducted by the investigators was designed in 1983–1984 (Flynn et al., 1992; 1994; 1995). Youth smoking rates (thirty-day smoking among grade 12 students) had stabilized at about 30% after a decline from nearly 40% in 1976. Although youth smoking was considered a serious problem, well-organized tobacco control efforts were not a high priority. The public policy emphasis currently seen in the United States and other countries had not yet emerged. Thus the social context for youth smoking in the early 1980s could be characterized by concern about the problem, but minimal levels of investment in public education and public policy intervention.

Research on youth tobacco use, however, had accumulated a body of evidence with clear implications for design of interventions. Epidemiologic studies indicated that the ages of highest risk for experimentation and initiation of regular use of cigarettes were during the preadolescent and adolescent years. The literature of that time also provided strong evidence that initiation of smoking was influenced by social factors. These results led the researchers to test school-based cigarette smoking prevention interventions focused on the years of transition from childhood into young adulthood. Researchers also developed the social influence model for intervention based on social cognitive theory (U.S. Department of Health and Human Services, 1994); school-based programs using this model consistently showed initial reductions in the proportion of students who began regular smoking. However, when students were followed through grade 12 these effects typically were lost. It appeared that prevention effects achieved at one point during adolescence could be overwhelmed by pro-smoking influences that operated throughout these developmental stages.

Interest in finding ways to build on the modest success of school smoking prevention programs led to consideration of alternate strategies. The literature lacked knowledge about the impact of mass media campaigns on young people’s decisions about smoking. The background literature on media effects in general and the strong media presence of the tobacco industry, however, suggested that a large dose of media intervention could have a significant effect on youth smoking behavior. Although school-based smoking prevention programs were not widely used in schools at that time, successes with the social influence model suggested that similar programs would eventually be adopted throughout the U.S. and that the relevant question for media research was whether that approach could add further impact to an effective school program.

These considerations led the investigators to propose that a multi-year mass media campaign based on the social influence model could be used to broaden and extend the impact of the school-based prevention programs. With the resource limits proposed by the funding agency it would not be possible to support a set of mass media campaigns simultaneously addressing adolescents at different developmental stages. The solution was to focus on one age-cohort of youngsters, providing age-appropriate interventions as they matured over four years. The purpose for this design choice was to apply intervention resources in as powerful a way as possible to a single group of young people over an extended period of time. The overall hypothesis asserted that regular cigarette smoking would increase less among adolescents receiving the combination of a mass media intervention and a school smoking prevention program than among adolescents receiving the school program only. Additional hypotheses concerning effects on mediating variables were based on social cognitive theory (Baranowski, Perry, & Parcel, 2002).

Intervention plan for the Media-School Study

The project created the mass media and school intervention components using the social influence conceptual framework. Both components were implemented in experimental communities; but just the school component was implemented in comparison areas. The media intervention consisted of radio and television spots targeted to a cohort of youths over a four year period, generally during grades 6–9 (ages 12–15). The plan provided for insertion of new media messages annually because of the expected need to refresh campaigns during this long period, and because of differing needs and interests of young people at different stages of adolescence. Message development was organized through a structured process that considered the perspectives of audience samples, media producers, and a range of expert reviewers (Worden, et al., 1988). The first year’s television and radio campaign ran primarily from January through May 1986. About 15 television spots and 8 different radio spots were broadcast during each of the 4 years.

Message placement and campaign weight

Messages were broadcast primarily as paid ads over local media. Placement was purchased in channels and programs that were indicated as favorites in annual school surveys. Purchased time was supplemented by donated time matches at approximately 50% of the purchased time. Placement of paid exposures in specific broadcast programs or viewing periods was guided by survey data. Weekly Gross Rating Points (GRPs) during January–May and August–September for TV and radio combined was estimated at 380, or 3–4 exposures per week. During June–July, radio-only campaigns delivered approximately 215 GRP weekly.

School program

To provide a uniform school smoking prevention program, the investigators developed a program for grades 5 to 10 that included grade-specific curricular materials, annual teacher training, and monitoring of implementation. The curriculum was delivered by usual classroom teachers who were trained by project staff. The school program was linked with the media intervention only at the level of educational objectives.

Study design for the Media-School Study

This study used a quasi-experimental controlled design that provided for repeated measurement of hypothesized outcomes over a five year period among the student cohorts in two study conditions. Cigarette smoking status and targeted mediators of smoking were assessed in five annual classroom surveys in both conditions. In the experimental condition students were exposed to mass media and school-based interventions for four years. In the comparison condition students were exposed only to the school program for four years. The study was conducted in four mass media markets with two markets assigned by the investigators to each study condition. The participants were in grades 4, 5, and 6 at the baseline measurement in spring of 1985, before any interventions were introduced. They then participated in followup classroom surveys each spring from 1986 through 1989, the years during which interventions were implemented. Annual survey responses were linked longitudinally for individual participants. By the end of the main study period these participants had advanced into grades 8, 9, and 10 (see Figure 1). Supplementary funding permitted extension of this design to conduct an additional set of surveys two years after completion of the original study when the cohorts were in grades 10, 11, and 12. Although not originally designed with consideration of clustered sampling, data were analyzed with techniques that satisfied the requirements of this design feature (Murray, 1998).

Figure 1.

Research design for Media-School study (1985–1989).

Selection of media markets

Two pairs of media markets were selected as best matches from regional pools. Selection criteria included being an independent market and having a population between 50,000 and 400,000. Within these constraints, pairs of markets were selected by matching on key demographic characteristics associated with cigarette smoking prevalence; these included similar levels of adult educational attainment (percent graduated from high school and percent with four-year college degree), median household income, and race/ethnicity distributions as reflected in 1980 U.S. Census data. The selection process began with the home market for the investigators in Vermont; a good demographic match for that market was identified in the northeast region of the U.S. and a matched pair of markets was identified in a western U.S. state. Study samples were matched more closely through selection of specific school districts and feeder school units within these four markets on the basis of demographic data from census tracts for the localities serving as catchment areas for the schools. Following recruitment of school districts and parental consent procedures, a cohort of 5458 students from these four media markets was established in grades 4, 5, and 6 in the spring of 1985.

Assignment to study condition

Within one community pair, the media-and-school community was so designated because of a need in the paired community to avoid contaminating other health promotion studies. In the second community pair, the media-and-school community was so designated because its media served fewer outlying communities and broadcast costs were lower. Based on these considerations, the four media markets were assigned by investigators to study condition within the matched pairs.

Data collection and measurement

All data were collected through classroom surveys conducted by research project staff at approximately the same time each year in the participating schools in all four media markets. Paper surveys were coded with unique identifiers that enabled linking of individual responses across survey years. Intervention effects were assessed using standard smoking behavior measures and theory-based assessment of psychosocial mediators. Mediator measures were based on the educational objectives used to create messages; content validity of mediator measures was confirmed using factor analysis methods. Measures of cigarette smoking status included the Smoking Behavior Index which estimated number of cigarettes smoked per week, and individual measures of smoking in the past day and past week. Mediators targeted by the interventions included Attitude Toward Smoking, Advantages of Smoking, Disadvantages of Smoking, Smoking Norm, and Perceived Smoking By Peers; these multi-item scales demonstrated adequate to high internal consistency reliability.

Results of Media-School Study

Baseline comparisons generally showed no differences between study groups for outcome or mediator variables. Consistent patterns of effect favoring the media-school condition emerged over time for smoking behaviors and mediators of these behaviors targeted by the educational interventions. At grades 8–10 about 13% of the media-school cohort and about 20% of the school-only cohort reported smoking cigarettes in the past week. Significant differences between the two cohorts were found at the end of the main study for all targeted mediators. The 35% relative reduction in weekly smoking prevalence in grades 8–10 was maintained over the additional 2-year follow-up period as shown by results of the grades 10–12 surveys (see Figure 2).

Figure 2.

Impact of media interventions on smoking prevalence in Media-School study. Weekly smoking results shown for students receiving the Media plus School (MS) and School only (S) interventions.

The interventions had consistent effects only on targeted variables. Non-targeted behaviors (alcohol and smokeless tobacco use) and non-targeted psychosocial variables (stress, perceived adult tobacco use) showed no significant intervention impact, suggesting that the effects were independent of underlying differences in the study groups. The consistency of media impact also was seen for smoking behavior when the two pairs of communities were viewed as four separate communities. The two mass media intervention communities showed a clear separation from the two school-only communities in the third through the fifth surveys (Flynn et al, 1992); similar patterns were seen for mediating variables. Further evidence for internal consistency of results was seen by comparing the temporal ordering of the appearance of differences among the targeted variables; in keeping with theoretical expectations, differences generally appeared sooner in mediating variables and later in smoking behavior variables.

Policy Impact of Media-School Study

This study had a substantial impact on policy recommendations in the years following publication of reports. The media intervention was seen as innovative because of a tested conceptual framework, intensive audience research, message development and placement focused on audience segments, and high intensity and duration of exposure. The study was notable also for the size and internal consistency of the effects. The controlled design was a feature that appeared to establish a high level of credibility for the results among policy makers. The study was featured in multiple reviews and policy statements. These included reports of the U.S. Surgeon General (U.S. Department of Health and Human Services, 1994), a report from the U.S. Task Force on Community Preventive Services (Task Force on Community Preventive Services, 2001), and early editions of the Centers for Disease Control and Prevention statement on Best Practices for Comprehensive Tobacco Control Programs (Centers for Disease Control and Prevention, 1999). Use of mass media for smoking prevention and other tobacco control purposes expanded significantly in the past two decades, as reflected in a recent comprehensive review (National Cancer Institute, 2008).

A Randomized Controlled Trial of Mass Media, 2000–2005

A second population-level experimental trial of mass media effects was conducted by many of the same investigators in 2000–2005. The youth cigarette smoking context had changed dramatically in several respects over the preceding decade. During the 1990s youth smoking rates increased from a low of about 28% of 12^th graders smoking regularly to almost 37% in 1997. This sharp increase was attributed to increased industry efforts to target younger smokers to counteract declining sales among adults, and to lack of substantial investment in tobacco control programs and policies.

The climate for further development of mass media strategies was positively influenced by a combination of concern over rising youth smoking prevalence, wider recognition of the serious consequences of cigarette smoking, and confidence that well-focused policies and programs could have positive effects. By the late 1990s good evidence had accumulated that higher taxes, restrictions on smoking in public places, and similar tobacco control policy measures could have a favorable impact on smoking rates and exposure to secondhand smoke (Farrelly, Pechacek, Thomas, & Nelson, 2008).

Simultaneously, extended experience with statewide tobacco control programs in California and other states indicated that large scale intervention programs were feasible, acceptable, and probably effective. By this time increasing numbers of school districts had adopted a substance abuse curriculum for the adolescent grades that included a focus on prevention of tobacco use, and often was based on a framework consistent with the social influence model. The successful multi-state lawsuit against the tobacco industry in late 1998 provided for large payments to states and national organizations that could be used, at least in part, for tobacco control programs. The climate for youth smoking in 1998–1999 when this study was planned could be characterized as a time of high levels of concern about youth smoking, substantial focus on smoking in public policy, statewide programs, and schools, and the prospects of continued public policy development and larger-scale investment at the national and state levels in tobacco control programs.

Based on this assessment, the investigators designed a youth tobacco intervention that would represent a well-funded campaign at the national or state level. The intervention consisted solely of mass media messages since schools were generally adopting substance use curricula addressing tobacco use. Based on prior success with the social influence model, and supported by reviews of media interventions that provided no clear indications that alternate message strategies were more effective, this campaign was based on the same conceptual framework used in the previous study. Thus the PRYSM (Program to Reduce Youth Smoking through Media) study planned a randomized controlled trial to test the impact of a public health model for mass media interventions alone on cigarette smoking prevalence in youth populations. The overall hypothesis stated that regular cigarette smoking would be lower among populations of adolescents receiving targeted mass media campaigns than among those not receiving these campaigns. Hypotheses concerning mediators were based, as previously, on the social cognitive theory framework that provided a basis for message design and measurement of effects.

PRYSM mass media intervention design

The multi-faceted media intervention consisted of four separate campaigns targeted a different age groups and conducted simultaneously during four years in the intervention markets. Each campaign consisted of television and radio messages (30-seconds long) developed for these campaigns placed on cable and broadcast television and radio programming favored by the targeted audiences. All messages addressed the intervention objectives and were not linked to other interventions in schools or communities. The target populations (grades 4–12) were segmented into three age groups Grades 4–6, 7–8, and 9–12 – because of the strong impact of age on youth preferences for messages and programming. Three prevention campaigns (one targeted toward each age group) were conducted from 2002–2005, and a fourth campaign targeted Grades 9–12 smokers-only during 2002–2004. A common set of educational objectives consistent with the hypotheses guided the four campaigns.

New messages were developed annually for each of the four simultaneous campaigns using systematic processes similar to those used in the prior study (Worden, et al., 1988). A diverse group of 15 production companies developed message concepts based on the educational objectives and briefing materials derived from focus groups representing targeted audience segments; concepts were reviewed by panels of social scientists and media experts. Selected concepts were developed and tested with diverse youth audience samples. Messages generally represented the major racial/ethnic groups receiving the campaigns, with about one-third of the messages reflecting a Hispanic casting and lifestyle, one-third African American, and one-third Caucasian. Multiple formats included dramas, comedies, testimonials, and cartoons. Approximately ten messages were chosen for each campaign in 2002 based on high endorsements by test audiences and balanced coverage of objectives and audience segments. Five additional messages were developed annually for each campaign in 2003–2005 using similar methods, to replace messages with declining appeal based on annual monitoring surveys.

PRYSM message placement and campaign weight

Messages were placed using purchased time in intervention community media accessing the largest number of youth at each age level. Results from annual small-scale Monitoring Surveys were used to select programming rated highest by each age group, gender, and race/ethnicity sub-segment in each market. Specific television channels and programs and radio stations used by each gender and race/ethnicity group at each age level were selected in each market to maximize exposure to messages for youth having these characteristics. Message placement plans were revised annually to respond to changes in media availability and youth preferences. Considering exposure to any of the three prevention campaigns, viewers were estimated to have received an approximate weight through TV and radio of 380 weekly GRPs during January–May and August–September and approximately 215 GRP weekly from radio-only messages during June–July; similar levels were estimated separately for exposure of the targeted groups in the cessation campaign aimed at smokers only.

PRYSM study design

The PRYSM study used a group-randomized experimental design based on communities (media markets) as the unit of assignment to condition and youth residing in those communities as the units of observation. Summative assessments were conducted before and after the four year interventions with separate cross-sectional samples (see Figure 3). Statistical power analyses based on this design and anticipated intervention effects specified the numbers of media markets and survey participants needed to provide acceptable power. Four demographically-matched pairs of media markets were identified in four states. One community was randomized to the experimental condition and one to the comparison condition within each pair. Experimental condition students in grades 4–12 were exposed to four years of media campaigns targeted to specific age groups; comparison condition students were not exposed to any interventions from this study. Based on the clustered sample design the required number of student participants in the baseline and follow-up surveys was 2,500 in each of the 8 media markets for each of the two summative surveys. This sample size provided 85% power to detect a 6% net difference between the intervention and comparison communities in changes in smoking prevalence using a one-tailed test at a 5% (p<.05) significance level. Data analysis plans were based on the community as the unit of randomization to study condition.

Figure 3.

Research design for PRYSM study (2001–2005).

Selection of media markets and randomization

Media market selection emphasized geographic and racial/ethnic diversity. Four matched pairs of medium sized metropolitan areas with similar independent television and radio media resources were identified in four states (Florida, South Carolina, Texas, and Wisconsin) based on the demographic criteria used for matching in the earlier study. The eight markets also were chosen to include substantial representation of African American, Hispanic, and Non-Hispanic White youth in Grades 4–12. Recruitment of school districts within these markets focused on lower income and education populations. One member of each pair was randomized to receive the media interventions after the 2001 baseline survey was completed.

Data Collection and measurement for PRYSM

Cross-sectional summative surveys were conducted in Grade 7–12 classrooms in all eight markets in 2001 and in the same schools in 2005; each survey included about 20,000 students. These Grades were chosen for evaluation because they were judged most likely to indicate detectable differences in smoking rates. A cross-sectional rather than a longitudinal survey approach was adopted to match the intervention plan targeting multiple fixed grade categories over time, rather than focusing on one age cohort as it matured. This approach also was favored by feasibility considerations including the potentially high cost of conducting longitudinal measurement with such a large sample, and a reduced tolerance of schools for conduct of multiple classroom surveys for research purposes that emerged during the interval between the two studies. A single-item measure of 30-day smoking prevalence used in the national Youth Tobacco Survey assessed smoking status. Targeted mediator measures assessed Perceived Community Peer Smoking Prevalence, Perceived U.S. Peer Smoking Prevalence, Peer Smoking Norms, Confidence in Refusing Cigarettes, Negative Outcome Expectations, and Positive Outcome Expectations; the internal structures of these multi-item scales were validated through factor analyses and demonstrated high reliability.

Potential impact of PRYSM

Reports of results from this study are in the process of publication review and revision, and will be available soon. It is likely that these results will have an impact on future thinking about the appropriate role of mass media in youth smoking prevention programs. The use of a powerful design and a large, highly diverse participant population are qualities that will lend weight to the importance placed on the results. The long cycle for publication of research reports, accumulation of evidence, and synthesis and analysis of multiple sources of evidence by researchers and policy makers suggests that the impact of this study may best be understood a decade or more after its completion.

Notable Features of The Two Controlled Trials

A comparison of key features of these two research studies may highlight important issues in designing such studies. Several strengths these studies had in common regarding both internal and external validity should be noted. These research designs both featured matched comparison groups that received the same schedule of measurement that the intervention group received; in both studies participants in the two conditions were shown to be equivalent on the outcomes of interest at baseline. Both studies were population-based, that is they targeted populations of adolescents in entire communities. Within these communities, both focused on those at higher risk for smoking. Several common strengths of the interventions also should be noted: a focused conceptual basis; targeting to segments through audience research; multiple messages provided through popular media; purchased air time to ensure appropriate placement; substantial campaign intensity; and duration over four years of high smoking initiation risk.

Unique Features of the Media-School Study

In addition to the common strengths for both studies, the Media-School Study had several advantages that were unique to the environment in which it was conducted. Both the mass media and the school interventions were unprecedented and unrivalled as intensive programs focused on tobacco use prevention in the study communities in the late 1980s. The educational vacuum into which these innovative programs entered may have provided the advantage of novel and highly relevant messages for these youth audiences. The media environment of that era provided an additional unique advantage because of the relatively narrow set of media entertainment choices that were available to young people; major network entertainment still dominated the available choices in a way that it no longer does, consequently simplifying the message exposure task. The statistically powerful cohort (i.e., panel) feature of this study was feasible because of the relatively small sample size (about 5500) and because of the tolerance of schools for repeated research surveys at that time. Neither of these conditions would be easily replicated for a similar study today.

This design had several disadvantages. The foremost issue was the lack of randomization of communities within matched pairs to study conditions. Also, the rationale for introducing a school intervention into both experimental and comparison communities has been presented, but the presence of this standard element in both conditions can create difficulties in interpreting results due to having no media-only or control conditions. The overall size of the trial has been viewed as small in comparison to state-wide programs; under contemporary design standards a mass media experiment based on four communities is unlikely to be proposed because of low statistical power. The specific communities chosen for this study also can be seen as a concern because of the lack of racial and ethnic diversity in the populations of these communities at that time. Although this design raised some issues of both internal and external validity, the strength and internal consistency of the intervention effects have been widely accepted.

Unique Features of the PRYSM Study

The PRYSM study has strengths in both internal and external validity that distinguish it from its predecessor. Foremost of these is that it was a group randomized controlled trial, with matched pairs of communities within four states randomized to study condition; this strong experimental design had sufficient statistical power to detect moderate effect sizes. In addition to the larger size of the study (eight markets vs. four) the selected communities provided large and highly diverse participant samples that were more representative of the populations that would be addressed by a national campaign.

This study faced some challenges in both its design and in the environment in which it was conducted. The study design provided for two summative surveys, one at baseline before the interventions were introduced and one after the four year interventions were completed. Small sample monitoring surveys were conducted in the interval to assess media use and exposure, but outcome data were not collected in these surveys. Under these circumstances it was not possible to track media effects while the interventions were implemented. Better tracking of outcomes combined with a more flexible intervention protocol may have provided additional opportunities to follow changing youth audience media patterns. In addition to significant changes in the youth media environment, this study also was implemented in an environment where multiple school, community, state and national tobacco control programs competed for the interest and attention of study participants.

Concluding Observations on Controlled Trials and Mass Media

The two studies were designed and conducted by teams of public health researchers motivated by a common need to address gaps in knowledge identified at the time each study was developed. One study focused on establishing that a mass media campaign could have an impact on decisions by young people to avoid tobacco in an environment characterized more by pro-tobacco than anti-tobacco influences. The second study focused on whether a multi-faceted public health media campaign could affect smoking prevalence in diverse youth populations in an environment characterized by many anti-tobacco influences. The perceived importance of the tobacco problem in each of these environments affected the resources available to conduct these studies and the specific design chosen. The earlier study was designed within well-defined budget limitations; the narrowly focused set of study areas and the small number of study communities facilitated a research plan that fit within these restrictions, but would not be considered adequate from a design perspective today. The stronger, and more costly, design features of the second study were made possible because funders were more willing to invest in a study for which importance was perceived to be very high and for which a strong record of preliminary studies was available.

These experiences lead to several observations about conditions under which the substantial investment required for conduct of a mass media campaign evaluation using a randomized controlled trial design might be warranted. The key issue related to choice of an evaluation design concerns the purpose of the evaluation. Development of new knowledge depends on utilization of a research design that can be widely recognized as credible because of the potential scope of policy and program decisions that may be developed based on the knowledge generated. The importance of the new knowledge generated by a field experimental design can be judged by the characteristics of the problem addressed. The issue addressed should be a high priority, long-term problem that has a clear-cut behavioral component that the media could plausibly influence to a detectable extent. Several other criteria also might be considered. Strong preliminary evidence should be available demonstrating that the message strategy is likely to have an impact on the behavioral outcome. Exposure plans for the targeted groups should be credible. The design should provide for tracking of outcomes to provide sufficient information to assess the intervention effects in mid-stream. The intervention protocol should provide sufficient flexibility to make adjustments in message content and placement as audiences and media change. Potential competing influences that may diminish differences between study conditions also should be assessed to ensure that effect sizes detectable by the design can be achieved.