Using Theory to Design Evaluations of Communication Campaigns: The Case of the National Youth Anti-Drug Media Campaign

Abstract

We present a general theory about how campaigns can have effects and suggest that the evaluation of communication campaigns must be driven by a theory of effects. The National Youth Anti-Drug Media Campaign illustrates both the theory of campaign effects and implications that theory has for the evaluation design. Often models of effect assume that individual exposure affects cognitions that continue to affect behavior over a short term. Contrarily, effects may operate through social or institutional paths as well as through individual learning, require substantial levels of exposure achieved through multiple channels over time, take time to accumulate detectable change, and affect some members of the audience but not others. Responsive evaluations will choose appropriate units of analysis and comparison groups, data collection schedules sensitive to lagged effects, samples able to detect subgroup effects, and analytic strategies consistent with the theory of effects that guides the campaign.

Evaluations of communication campaigns provide an opportunity to improve interventions as well as conduct research that is theoretically interesting (Valente, 2001). As public communication campaigns continue to grow in scope and level of sophistication (Lapinski & Witte, 1998), the task of evaluating campaigns becomes increasingly complicated. Decisions about the standards against which to measure campaign success, strategies for separating campaign effects on outcomes from those of other sources of influence, and expectations for differential campaign effects across subpopulations are only a few examples of the complexity faced by researchers who seek to evaluate communication campaigns (Hornik, 2002). Many researchers agree that theory-based evaluations are essential for dealing with this complexity (see Rice & Atkin, 2001). Besides guiding all stages of the evaluation from the formative study to the interpretation of results, theories provide powerful tools for a systematic inquiry of processes, such as behavior change in response to campaign messages, that are essentially dynamic and multifaceted.

Still, inadequate or incomplete theorizing may lead researchers who conduct evaluations of communication campaigns to erroneous conclusions concerning campaigns’ influence on targeted behaviors. Poor campaign theory may lead evaluators to look at the wrong outcomes, may lead them to expect behavior changes prematurely, or may lead them to use the wrong units of analysis or make comparisons between inappropriate groups. For example, many evaluations of communication campaigns attempt to demonstrate an association between direct individual exposure to campaign messages and rapid change in individual cognitions (e.g., attitudes, beliefs, perceived self-efficacy) and behavior (Lapinski & Witte, 1998). Oftentimes, this line of inquiry results in inconclusive or no evidence of campaign effects (Atkin & Wallack, 1990; Brown & Walsh-Childers, 1994; Hornik, 1997; McGuire, 1986). The failure to find effects can reflect a true failure of the campaign because of poorly chosen behavioral objectives, poorly designed messages, or, quite often, because of insufficient exposure to campaign messages.

The failure, however, may also reflect inadequately theorized and thus inadequately realized evaluation design. The effects of a particular campaign on behavior may occur only after some delay, or be small and undetectable with the small samples that are typically available. For example, antitobacco efforts have produced a sea change in smoking behavior over 40 years, but reductions have been 1–2% a year (Warner, 1981). Also, effects may be restricted to a particular audience. For example, safe-sex promotion campaigns have shown substantial success, but only when the samples studied focused on young people engaging in casual sex.

Similarly, evaluations that focus on the wrong outcomes may miss important effects. Although there is good evidence for HIV/AIDS campaign effects on condom use, there is very little evidence for short-term effects on other safer sex behaviors, such as reductions in numbers of partners among heterosexual populations (Wellings, 2002). The impact of communication campaigns may go beyond individual cognitions and behaviors to include effects on communities, institutions, organizations, and social networks. For example, anti–drunk-driving campaigns may have much of their effect through their influence on changes in public policy rather than through direct effects on drunk-driving behavior (Yanovitzky & Bennett, 1999; Yanovitzky & Stryker, 2001). If this is the case, evaluations that look for evidence of effects by comparing individuals who vary in personal exposure to anti–drunk-driving messages will not find such effects. In each of these cases, failure to match the evaluation design with the theory of the program will likely result in underestimating the success of communication campaigns.

The goal of this article is to present some aspects of a general model of media campaign influence on audience behaviors that will serve as a useful framework for designing systematic and rigorous evaluations of communication campaigns. We begin by presenting the model and outlining the theoretical rationale behind the different routes of campaign effects conceptualized. Important methodological implications of the model are discussed as well. We then apply this model to the evaluation of the current national-scale antidrug media campaign. There are aspects of a general campaign effects model that we do not address in much detail here. In particular we set aside issues related to the design of persuasive messages, and we offer a model that complements, rather than replaces, established models of individual behavior change (e.g., theory of reasoned action, health belief model, and social cognitive theory). In other words, we focus on the broader theory of campaign effects, taking the construction of individual messages and individual processes of behavior change as a given. We discuss the implications of the components of the broad model for evaluation design.

Aspects of a General Model of Media Campaign Influence

A theory of effects for a particular campaign needs to reflect answers to each of the following questions: What are its routes of effect: individual, social, or institutional? What is the expected lag between initiation of campaign exposure and effects? What is the nature of expected outcomes? Are effects expected to vary across subpopulations? How much exposure is needed before effects can be expected? Are effects dependent on exposures across channels over time? We present each of these somewhat overlapping questions in turn, with some emphasis on the first of them.

What Are the Routes of Effect?

As Figure 1 illustrates, there are at least three general paths through which media campaigns may influence behavior. The most obvious one involves direct exposure of individuals to the persuasive messages generated by the campaign, whether through ads placed in the media, educational programs, or other forms of messages. As a result of this exposure, individuals may learn about the costs and benefits of performing the behavior targeted by the campaign and form, accordingly, attitudes and beliefs regarding this behavior. They may learn what the social norms are for performing the behavior. They may also acquire skills necessary to perform or avoid a certain behavior and gain the self-efficacy to do so. Consequently, they develop positive or negative behavioral intentions that are eventually translated into an actual behavior. This path of media effects is derived directly from influential theories of health behavior change, such as the theory of reasoned action (Fishbein & Ajzen, 1975), the theory of planned behavior (Ajzen, 1985), the health belief model (Becker, 1974; Rosenstock, 1990), and the social cognitive theory (Bandura, 1986). It is also the path of effects most commonly conceptualized and tested in the design and evaluation of many communication campaigns. Typically, when an evaluation compares individuals who are more and less exposed to messages, it is assuming this individual model of effects.

Figure 1.

Overall Model of Communication Campaign’s Influence

The two remaining general paths of campaign effects are not as obvious as the direct effect path because they pertain to processes that occur at the higher than individual level of aggregation. The first of these concerns the diffusion of campaign themes to other social institutions, such as the mass media, the executive and legislative branches of government, the justice and law enforcement system, and religious organizations (see Yanovitzky & Bennett, 1999). Many campaigns, for example, include a media advocacy component that is designed to attract media and policy attention to the campaign goals (Wallack, 1990). In other cases, campaign messages attract institutional attention (e.g., Rogers, Dearing, & Chang, 1991).

Once the campaign themes have diffused to institutions, their influence on individuals may take several forms. It may be that people will increase their exposure to the campaign messages through their interaction with social institutions and organizations. For example, people who were not directly exposed to the campaign may learn about the risk of abusing alcohol through their interaction with religious leaders in the community (i.e., secondary exposure). It may also be that actions taken by institutions and organizations in response to the campaign activity will provide individuals with an additional source of motivation to change in the desired direction. Thus, newspapers may be convinced of the importance of drinking and driving in the presence of an active public service announcement campaign or as the result of media events staged by advocates. Subsequent critical newspaper coverage may reach many drivers who are discouraged from mixing drinking and driving. Finally, institutions and organizations can directly influence the behavior targeted by the media campaign by placing external constraints (legal, economic, physical, etc.) on undesired behaviors or creating incentives for involvement in desired practices. Examples include limiting alcohol availability to youth by setting a minimum drinking age (DeJong & Hingson, 1998), increasing taxes on cigarettes to reduce smoking (Hu, Sung, & Keeler, 1995), and providing women with free access to mammography screening (CDC, 1997).

If individual behavior is affected through campaign effects on institutions rather than only because of direct exposure to messages, an evaluation that relies on comparisons between more and less exposed individuals may miss the effects. For example, Yanovitzky (2002) found little direct association between media coverage and individual beliefs about drunk driving, but a clear association between aggregate media coverage of the issue of drunk driving and subsequent legislation to curb drunk driving, as well as an association between the passing of such legislation and the subsequent reduction in drunk-driving behavior and fatalities. If he had tried to evaluate the success of efforts to put drunk driving on the media agenda by looking for effects on public beliefs and behavior, he would have inferred there were no effects. Only the examination of the indirect path through policy implementation showed such effects.

The third general route of influence relates to campaign-induced processes of social diffusion. Through social interaction with family members, peers, and other members in the community, people learn about practices that are socially approved and those that are not. They also become aware of the costs and benefits, in social terms, of performing each behavior. This social information, in turn, helps to shape their behavioral attitudes, beliefs, and intentions.

It follows that media campaigns may influence their audiences through social diffusion in two ways (see Yanovitzky & Stryker, 2001). First, there may be simple secondary diffusion of messages. The campaign prompts those who are exposed to transmit its messages to those who were not exposed or attentive to the same messages. Second, the campaign may stimulate discussions among family members, peers, and other members of the community through which social norms and expectations concerning the underlying behavior are clarified. In both cases, the characteristics of one’s social networks (e.g., size, degree of cohesiveness, strength or weakness of ties to others, and the stability or instability of these characteristics over time) will either complement and amplify the transmission of campaign messages to individuals or hamper persuasion and social influence efforts (Rice, 1993).

Depending on these structural characteristics, then, diffusion of social information generated by the campaign may motivate individuals to make (or resist) the changes advocated (i.e., acquire skills, gain self-efficacy, or change attitudes and beliefs). It follows that in the presence of social diffusion, comparisons of those who are personally exposed with those who are not personally exposed may miss important effects. For example, in the evaluation of the National Youth Anti-Drug Media Campaign described below, 48% of all youth indicated they talked about the antidrug ads with their parents, siblings, or friends, and 55% of parents claimed to have talked about the ads with their children. In that circumstance, an estimate of exposure to antidrug advertising might justifiably include both types of exposure, and self-reports of direct exposure to such ads might underestimate the indirect exposures through word of mouth. In some cases it will be appropriate to use units of analysis larger than the individual (the family, the friendship group, the community) if those are the effective units for a theory of effect. In general, an association assessed at the individual level will underestimate effects of exposure that are shared within social networks.

What Is the Expected Lag Between Initiation of Campaign Exposure and Effects?

Some campaigns may expect immediate effects. There is good evidence, for example, for a surge in clinic visits in developing countries after family-planning campaigns announced that contraceptives were newly available at local clinics (Piotrow et al., 1990; Rogers et al., 1999). It appears that new knowledge solved a problem already perceived by those who were ready to change. Many other campaigns, however, may not foster quick responses among much of the population. Campaigns may expect delayed responses because they are dependent on a social or institutional model of message diffusion rather than a direct-learning model as described above. Typically, social or institutional diffusion would be expected to take longer than individual persuasion. Campaigns may take longer because they address behaviors with a deeper social or cultural anchoring, and individuals will take repeated convincing before they are ready to change. Few evaluations of family-planning communication campaigns that look for widespread change in the general population and that include many individuals not ready to change have shown such detectable effects (Hornik, 2002.) Effects may take longer to appear because the opportunities to engage in a new behavior are themselves delayed: Antitobacco education in elementary school, for example, is designed to inoculate youth against initiation of tobacco use years later when they are vulnerable.

Evaluations that measure outcomes too soon may easily come to the wrong conclusion about effects. As an example, Worden and Flynn (2002) found that differences in weekly smoking between youth in a mass-media-plus-school intervention versus those in a school-only condition were 0.6% for youth in fourth through sixth grade (1% versus 1.6%). However the gap grew to a highly significant 7.6% (16.5% versus 24.1%) among the same youth when they had reached grades 10 to 12, 2 years after an extended intervention had ended. If the researchers had looked too early for final results they would have come to a much less favorable conclusion.

What Is the Nature of Expected Outcomes?

Sometimes campaigns have a well-focused behavioral objective: They want, for example, to increase condom use at last vaginal intercourse by 25% among 18–35-year-olds who are having casual sex. Under that expectation, the evaluator knows exactly what to measure. However, sometimes the objective of a campaign is satisfied by the adoption of any of a set of behaviors. A campaign designed to lower birth rates may be successful regardless which of a long list of temporary or permanent contraceptive approaches couples choose. Similarly, a campaign to encourage environmental responsibility may expect that the adoption of that attitude will lead to a wide range of pro-environmental behaviors, depending on the individual’s context. A campaign may even focus its messages on one behavior but expect that its target audience will generalize to related behaviors. For example, in the Philippines, a successful measles promotion campaign was meant to serve as a stimulus for increased compliance with all vaccinations among the parents of very young children (Zimicki et al., 1994). In each of these cases, the reported success of the campaign can vary with what outcomes the evaluators choose to emphasize. Estimates of the success of the Philippines intervention varied with the criterion the evaluators used. If they used on-time (by 12 months) completion of a full vaccination series as a criterion, it had remarkable success, increasing the proportion of children nationwide who satisfied this criterion from 32% to 56% in 1 year. However, if they focused on full vaccination by 24 months, success was less striking, a growth from 54% to 65%.

What Variation in Effects Is Expected Across Subpopulations?

It is rare that everyone in a population is included in the target population for a campaign. An evaluation that does not restrict its samples to those considered vulnerable to effects will likely err in its estimates of effects. Sometimes that specification is trivial: Women are the primary target audience for mammography promotion. There is little point in including men in the samples, although men may be at a small risk for breast cancer, and they may be an audience who can support women in their decisions. However, sometimes the definition of relevant target audiences is subtler. An evaluation of an antidrug campaign in Kentucky found clear effects, but only when they focused on “high-sensation-seeking” adolescents, who were the only ones at much risk of marijuana use (Palmgreen et al., 2001). For these high sensation seeking youth, the Kentucky intervention was associated with an overall drop of nearly 9 percentage points in past 30-day marijuana use. If all youth had been included, the drop would have been about half that size, largely because low sensation-seeking youth were not at risk of marijuana use at all. If the Kentucky investigators had studied the whole population, rather than the group of high sensation seekers whom they had specified a priori as the susceptible group, they would have concluded there was no effect.

In general, campaigns operate in a complex environment; evaluators must take into account the other factors likely to limit or magnify the effects of the campaign. These other factors include demographics, personality traits, characteristics of the social environment, and prior experience with the behavior, or similar behaviors, that may influence the target behavior, behavioral expectations, and intentions, as well as the propensity of being exposed to the campaign. From the evaluation perspective, it is prudent to identify and control for such confounding influences when estimating campaign effects. More importantly, the potential influence of exogenous variables provides ample justification for researchers to anticipate and study interaction effects, namely, the potential for differential campaign effects on distinct subpopulations.

How Much Exposure and How Much Time?

Simple information diffusion campaigns may be able to depend on a few exposures to a message through single channels. The theory of such a campaign, though, much like the contraceptive clinic example above, assumes that there is a ready audience for new information that solves an accepted problem. Such limited exposure might have been enough in the efforts that brought about a switch from aspirin to Tylenol to avoid the rare children’s illness Reye’s Syndrome (Soumerai, Ross-Degnan, & Kahn, 1992) because the information about the risk of aspirin was new, the change was very low cost, and the potential risks associated with the old behavior were large. This confluence of favorable factors, however, may not be so common. A more typical campaign may depend on operating through multiple channels, diffusion of messages repeatedly over time, and supportive actions from institutions and social networks. The operation of the National High Blood Pressure Education Program (Roccella, 2002), which was associated with massive changes in control of blood pressure and reductions in stroke rates, was typical of such “kitchen sink” campaigns. That campaign worked through professional organizations, through public service announcement programs and public relations efforts with local media outlets, and through any other means it could. A messy kitchen-sink campaign forces evaluators to give up randomized control groups and to look at units of analysis larger than the individual if the theory of the campaign is to be respected in the evaluation.

Summary

Each of the questions discussed above leads to the specification of a theory of the campaign. The theory of the campaign leads directly to the set of related methodological considerations. There are some evaluation designs that will be consistent with a particular theory of campaign effects and some that will not be. The evaluators will need to determine the appropriate unit (individual, social network, community) for expected effects, although they may decide to work at more than one of these levels. They will need to specify the appropriate lag between exposure and effects and whether the effects will slowly increase over time. They need to decide whether to focus on specific outcomes or on generalized effects. In the context of other factors that make individuals more or less vulnerable to campaign effects, the evaluators need to decide what subpopulations are to be the focus of the evaluation. They may decide whether the way the campaign is expected to have its effects is consistent with a controlled experimental design with deliberate manipulation of exposure levels, or they may decide that they can rely only on natural variation in exposure among individuals or larger social units if they are not to control away the important but messy kitchen-sink effects on which some programs may rely.

We turn now to an example of a major media campaign and present a theory of its effects and the evaluation design that was chosen to respond to that theory of effects. In some aspects the evaluation design was able to follow the strictures laid out above; in others, it had to compromise because of resource and other limitations.

National Youth Anti-Drug Media Campaign

In 1998, the Office of National Drug Control Policy (ONDCP) launched the National Youth Anti-Drug Media Campaign (hereafter, the campaign). The campaign’s specific goals are preventing and reducing drug use among American youth and encouraging parents to play a more active role in this process (primarily through monitoring and talking to their children). The campaign built on a long-term program by the Partnership for a Drug-Free America that depended on contributed labor and advertising broadcast time and print media space. The new campaign was launched with a 5-year appropriation from the U.S. Congress of nearly $1 billion so that it could purchase advertising time and pay for other components. The media campaign has progressed through three phases of increasing complexity and intensity. Phase I (January to June 1998) involved pilot testing the intervention in 12 metropolitan areas. In Phase II (July 1998 through July 1999), these advertisements appeared in multiple media nationwide, not just in the test areas, and new advertisements were added. Phase III, which began in Fall 1999 and is scheduled to continue at least through June 2003, marks the full implementation of the campaign and involves the dissemination of antidrug messages to a national audience of youth and parents. More detailed description of this media campaign appears elsewhere (see ONDCP’s website at www.whitehousedrugpolicy.gov).

The Theory of the Campaign

The campaign seeks to educate and enable America’s youth to reject illegal drugs, prevent youth from initiating use of drugs, especially marijuana and inhalants, and convince occasional users of these and other drugs to stop using drugs. It is the task of the evaluation to determine how successful the media campaign is in achieving these goals and to provide ongoing feedback useful to support decision making for the campaign.

Figure 2 begins to outline the theory of the media campaign. Only the model for youth behavior is presented, but it is worth noting that a similar model was developed for evaluating campaign effects on parental behavior. The model in Figure 2 portrays the routes through which the media campaign may influence youth drug behavior. Specifically, any one youth may receive antidrug messages from each of four sources:

Figure 2.

The Campaign Evaluation Model

1. Exposure to Media Messages. These messages may come from direct exposure to the media campaign advertisements on television, on the radio, in magazines, on the Internet, on billboards, and elsewhere. Also, youth may be directly exposed to unplanned antidrug media messages if, for example, the news media increase their coverage of the issue as the result of the media campaign activity. A youth’s likelihood of direct exposure to media antidrug messages depends on two factors: first, how often the youth is exposed to that communication medium (for instance, how often they watch television), and second, the number and nature of advertisements that are placed in a given time period and on that medium (usually measured as Gross Rating Points, GRPs).

2. Interaction With Friends and Other Peers. Antidrug messages may come from conversations with friends. These conversations among peers may have been stimulated by the presence of the media campaign, whether by advertisements or by activities undertaken by other organizations. It may be that the youth was involved in sponsored activities or saw the advertisements and introduced the topic; it may be that his or her friends saw the advertisements and introduced the topic. In either case, the campaign might have activated a social diffusion process that increased the number of drug-related messages heard by respondents. The media campaign may also stimulate discussion that rejects or reinforces antidrug messages. It is reasonable to expect that the prior attitudes of friends are an important influence on the valence of message retransmission. Thus, one might expect that talk among friends will result in the transmission of antidrug messages most often when the attitudes of friends are consistent with those messages. Similarly, the way in which youth respond to advertising messages will be influenced partly by the content of the advertisements but also by the way their friends interpret those messages. The model suggests that those interpretations will be partly dependent on the nature of friends’ attitudes. Individuals may see or hear the media campaign advertisements, but they may interpret them as antidrug messages or as preaching to be resisted, and thus made into prodrug messages, depending on the stance of their social network.

3. Interaction With Parents. Antidrug messages may come from parent-child conversations. One of the campaign’s early emphases has been to encourage parents’ involvement in their children’s lives and, in particular, to encourage conversations about drugs and drug use. If the mass media advertisements are successful, there should be more parent-child talk about drugs and thus a greater transmission of antidrug messages from parents to their children. In addition the campaign encourages other forms of parental involvement in their children’s lives, including active monitoring of their behavior and doing fun activities with them, under the theory that such involvement discourages initiation of drug use.

4. Interaction With Organizations. Partnership organizations, including general youth organizations (sports teams, scouts, and religious groups) and antidrug-focused institutions, are expected to increase their active transmission of antidrug messages. The media campaign intended to work through partnership organizations to encourage local action in support of the antidrug message. These organizations may reach enrolled youth directly or through parents or peers as intermediaries. The level of activity of partnership organizations may be affected by the presence of the media campaign.

Institutional message transmission may also come about as an indirect effect of the media campaign itself. The presence of the advertisements (and the other elements of the campaign) may produce a broad response among other public institutions, affecting the nature of what they do with regard to drug use. In turn, institutional actions may affect youth cognitions and social expectations about drug use and the youths’ own drug use behavior. Thus, campaign activities may stimulate concern about drug use among school boards and lead them to allocate more time to drug education. Religious, athletic, and other private youth organizations may increase their antidrug activities. News organizations may cover drug issues more actively, and the nature of their messages may change. Popular culture institutions (movies, music, entertainment television) may change the level of attention to and the content of drug-related messages.

The right-hand side of Figure 2 focuses on how exposure to antidrug messages might be turned into behavior. The model relies fundamentally on the basic approach of the theory of reasoned action (Fishbein & Ajzen, 1975) supplemented by the arguments of Albert Bandura (1986) concerning the importance of self-efficacy. The model assumes that intention to undertake an action is the primary determinant of whether that action will be undertaken, although external forces (e.g., the price of drugs, their availability, and the risk of arrest) may constrain or facilitate the transition from intention to action. The model assumes further that intentions are largely a function of three influences: attitudes toward specific drug behaviors, perceptions of how important others expect one to act, and the belief that one has the skills to take an action (called self-efficacy). The overall attitudes are in turn influenced by the youth’s beliefs about the expected positive or negative consequences of those behaviors. Perceived social expectations are a reflection of the specific expectations of each group of important others (parents, friends). Finally, the model assumes that exposure to antidrug messages will influence those beliefs, perceived social expectations, skills, and self-efficacy. In fact, there is good evidence that each of these predictors is an important correlate of drug use. For example, among nonusing youth, those who have high prodrug scores on these variables are more than 12 times as likely to initiate marijuana use in the subsequent 18 months as those with lower scores.

Figure 2 also provides a list of all of the external factors that are incorporated into the design of the evaluation. These include basic demographic characteristics and a broad category called family and peer factors. These factors include parental monitoring (which is a particular objective of the media campaign), family functioning, and friends’ attitudes and behaviors, including involvement with others who engage in risky behaviors—all of which have been shown to influence drug use and abstention (Bailey, 1989; Beck, Ko, & Scaffa, 1997; Botvin, Malgady, Griffin, Scheier, & Epstein, 1998; Brown, 1990; Dinges & Oetting, 1993).

The final category encompasses a range of personal factors, including sensation seeking, which, it has been argued, is an important determinant not only of drug use but also of responsiveness to advertising messages of a particular style (Palmgreen, Donohew, Pugzles Lorch, Hoyle, & Stephenson, 2001). In addition, the personal factors category includes academic success, ambitions, and religious involvement, which have been shown to predict drug use and abstention, as well as prior drug involvement, usually the best single predictor of future drug use (see Bailey, 1989). As discussed previously, all of these factors may directly influence any of the variables in the model, including who is and is not susceptible to the media campaign influence.

The model in Figure 2 cannot easily portray some other elements of the theory of the campaign relevant to the evaluation of campaign effects. First, it is possible that there will be time lags between the media campaign activities and their effects. Second, it is possible that messages directed toward a specific belief or behavior will generalize to other beliefs or behaviors. These elements are summarized below:

• Immediate Learning. As a direct result of the advertisements, youth immediately learn things about specific forms of drug use that lead them to make different decisions about those forms of drug use. For example, they learn that trying marijuana has bad consequences so they are less likely to try marijuana (but this belief does not generalize to other drugs). This new learning could have immediate consequences, which would be expected to show up in simultaneous associations of exposure with beliefs and behavior.

• Delayed Learning. As a direct result of the advertisements, youth learn things that lead them to make different decisions about drug use at a later time. The advertisements might have a delayed impact; their influence will show up immediately in associations between exposure and affected beliefs, but current exposure will predict only subsequent behavior. This might be particularly true for children, for whom current learning would be expected to influence future behavior, when opportunities to engage in drug use increase.

• Generalized Learning. The advertisements provide direct exposure to specific messages about particular forms of drug use, but youth learn things that lead them to make decisions about drug use in general. Thus, if they learn that cocaine has a particular negative consequence or that medical authorities are opposed to cocaine use, they may generalize those cognitions to a broad negative view of other types of drug use. From the perspective of the evaluation, this generalized learning would mean that exposure effects are not message specific and will not necessarily operate through an intervening path of acceptance of the specific consequences emphasized. This seems particularly likely among younger children, who may read the meta message of the barrage of advertisements as saying that drug use is bad, but without learning an elaborate set of specific rationales for that attitude.

The Evaluation Design

The evaluation design included each of the following major elements or characteristics:

1. Three discrete national samples of youths, ages 9–18, interviewed over three 6-month periods beginning around January 2000. The three samples totaled around 8,000 individuals, evenly divided among 9–11-, 12–13-, and 14–18-year-old age groups.

2. One parent, randomly assigned to be the mother or father, if appropriate, interviewed for each youth. Because some youths came from the same household, there were fewer parents—approximately 5,500—than youths.

3. These respondents represent the approximately 40 million youth and 43 million of their parents who are target audiences for the media campaign. The sample is designed to represent youth living in homes in the United States. The households were chosen from 90 different primary sampling units across the U.S. These primary sampling units fall within a single media market and will permit some comparisons across communities.

4. Each youth who remains in the target age range and his or her parent will be interviewed twice more at 12–18-month intervals until the end of data collection in June 2003.

5. Questionnaires are administered in respondents’ homes on touch-screen laptop computers. Because of the sensitive nature of the data collected during the interviews, a certificate of confidentiality was obtained for the survey from the U.S. Department of Health and Human Services, and confidentiality was promised to each respondent. Sensitive questions and answer categories appeared on the laptop screen and were spoken to the respondent in a recorded voice over headphones that could be heard only by the respondent. Participants responded by touching the laptop screen.

6. The questionnaire for youths included extensive measurement of their exposure to campaign messages and other antidrug messages. It also included questions about their beliefs, attitudes, intentions, and behaviors with regard to drugs and a wide variety of other factors either known to be related to drug use or likely to make a youth more or less susceptible to campaign messages.

7. The questionnaire for parents also included measures about exposure to campaign messages and other antidrug messages. In addition, it included questions about their beliefs, attitudes, intentions, and behaviors with regard to their interactions with their children. These included talking with their children about drugs, parental monitoring of their children’s lives, and involvement in activities with their children.

8. Ad exposure was measured for both youth and parents by playing current or very recent TV and radio advertisements for respondents on laptop computers to aid their recall. In addition, there were some unaided questions about recall of ads seen or heard on TV and radio and in other media, such as newspapers, magazines, movie theaters, billboards, and the Internet.

The evaluation thus has an explicit theory for the several ways that it might affect youth behavior. It also is collecting the data described in the previous paragraphs. How will the evaluation use this theory of effects to look at this data? In this discussion, we focus on how each of the major theorized routes of effect can be tested.

The first question is whether or not the youths and their parents were substantially exposed to the campaign and through what routes. Were they directly exposed to the campaign and did that direct individual exposure affect their drug-related cognitions and behavior? The core of the evaluation strategy to address this question has four elements: (a) It starts with the basic measurement of the specific exposure of each youth—on average over the first 2 years of the campaign, youth report seeing about one TV ad per week, for example, with some seeing more and some less; (b) next, the covariation of individual-specific exposures with outcomes, like intentions to use marijuana, is tested; (c) next, the possible role of confounders in accounting for any observed association is statistically controlled; and (d) finally, the concern that an observed association, even were it to hold up once confounders were controlled, might reflect the influence of the putative outcome on exposure rather than vice versa, is addressed by testing whether prior exposure covaries with lagged outcome, when the prior outcome measure is controlled. This analytic approach tests the most conventional of the routes to effect: whether direct individual exposure to the campaign affects individual outcomes.

A second proposed route of effect suggests that exposure is produced not only by direct exposure to campaign-produced advertising, but also because the campaign activates other channels. In one case, the other channel might be parents or siblings who are themselves exposed to the campaign and in turn influence the target youth. The test for influence through these family channels can be performed directly, because the evaluation design collects exposure data from one parent for each child and for a sibling in about half the families. It is then possible to examine the influence of parent or sibling exposure to advertising on focus youth outcomes, following the model described above for youth exposure on youth outcomes.

For other channels of potential influence the approach will be different. The campaign has intended to influence the activity of other institutions so that they discourage drug use, by providing antidrug education, for example. One test of these routes would be to show that the presence of the campaign had increased the level of such antidrug activity. Because the youths are asked about their participation in antidrug education in and outside of school, the trend in such activity over time can be traced. In addition, the covariation between youth participation in such activities and their drug-related cognitions and behavior, controlled for confounders, can be tested. If there is evidence both that such activities have increased over the course of the campaign and that there is some influence of that activity on desirable outcomes, tentative support for a campaign effect can be claimed. More confident claims that the campaign was responsible for increases in such activity, rather than some coincidental historical trend, might require additional evidence. If the campaign’s effects on institutional activity is assumed to result from heavy play of the ads affecting the willingness of institutional authorities to incorporate antidrug programming in their work, then an additional analysis might support a causal claim. It would be expected that the growth in institutional activity would be most notable in the communities where the ads were most often played. It would be possible to differentiate the 90 primary sampling units, which correspond most often to counties, according to their average ad exposure and their average institutional activity. If those were associated at the community level, after appropriate statistical controls, the case for a campaign effect on such activity would be strengthened.

It is possible that the effects do not reflect individual exposure by the youth to mass media or community institutional channels or influence reflecting the exposure of a parent or sibling. Rather, the effects might occur because of social diffusion of messages at the level of the peer network or the community. The design does not permit an easy test of the influence of a youth peer group on the individual youth (had the data been gathered from youth friendship networks, it would have been possible to test the covariation between the average exposure in the social network and the individual youth’s outcome measures). A more demanding version of this social diffusion hypothesis requires an assumption that the unit of effect for the campaign is the community (or the media market) approximated by the 90 primary sampling units in the study. If that were the case, and there was substantial variation across those communities in their exposure to campaign advertising, the association of such shared exposure with outcomes could be tested, incorporating information about average exposure at the community level. It would be appropriate then to look at a hierarchical model, to disaggregate evidence of any effects at the community level from effects at the individual level.

In addition to consideration of the unit of effect (individual, social, or institutional) the design will permit extended consideration of the process through which effects are generated. Figure 2 above laid out the pathways for effect; this drove the questionnaire design, which was meant to assess both the proposed intervening variables between exposure and behavior (beliefs and attitudes, social norms, self-efficacy, intentions) and the background variables, which might constrain or facilitate the influence of those variables on behavior. If associations between exposure and outcomes are established, it will be sensible to examine the intervening paths. Figure 2, for example, can be used to stand in for a path model, which can be used to assess how well the set of intervening variables accounts for the association of exposure and outcome. This analysis will be particularly intriguing because it can be carried out over three waves of measurement, allowing greater confidence in sorting out causal order among the three sets of variables (exposure, intervening variables, and outcomes). It also will be possible to directly test interactions between the background variables, particularly those known to put youth at risk of drug use, and the effects of exposure on outcomes.

Although the design provides many opportunities for testing all of the ways this campaign may affect its audience, including unexpected ways, it also has weaknesses. The design depends on natural variation in exposure for all of its claims of effect; if there is not enough variation (either there are too few people with minimum exposure, or not enough people with high exposure) potential effects may be undetected. Also, because exposure is self-selected, there is a risk, even with the implementation of sophisticated statistical controls, that observed associations are due to unmeasured factors rather than to campaign effects. Also, even though the ability to investigate lagged effects is a positive element of the design, its usefulness depends on the appropriateness of the chosen lag period (12–18 months).

Conclusion

We both present theory about how campaigns can have effects and suggest that the evaluation of communication campaigns must reflect that theory. We use the current evaluation of the National Youth Anti-Drug Media Campaign to illustrate both the theory of a campaign and what implications that theory has for the evaluation design and for the types of analysis of data appropriate, given how the campaign is expected to affect behavior.

Substantively we argue that many campaigns’ models of effect do not operate on the simple model that exposure will lead to new cognitions and that new cognitions will lead to behavior change among individuals over a short term, even though evaluation designs may act as if that were the only path of effect. Contrarily, effects may operate through social or institutional paths as well as through individual learning; they may require substantial levels of exposure achieved through multiple channels over time; they may take time to accumulate enough change to be detectable; they may produce effects on specific or on generalized outcomes; they may be expected to affect some members of the audience but not others.

In sum, we make two essential points: The way that campaigns can affect behavior is often complex, and if that complexity is not reflected in the evaluation design, many of the effects may go undetected. We show the nature of that complexity for one particular program and show how the evaluation design and proposed analysis approach respond to that complexity. The more general point is relevant to all such campaigns, however: Develop a theory of the campaign that respects how behavior can really be affected and evaluate the campaign consistent with that theory of effect.