Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2012 Jan 1.
Published in final edited form as: Commun Methods Meas. 2011 Jan 1;5(2):126–145. doi: 10.1080/19312458.2011.568374

Evaluating Mediated Perception of Narrative Health Messages: The Perception of Narrative Performance Scale

Jeong Kyu Lee 1, Michael L Hecht 2, Michelle Miller-Day 3, Elvira Elek 4
PMCID: PMC3148767  NIHMSID: NIHMS291724  PMID: 21822459

Abstract

Narrative media health messages have proven effective in preventing adolescents’ substance use but as yet few measures exist to assess perceptions of them. Without such a measure it is difficult to evaluate the role these messages play in health promotion or to differentiate them from other message forms. In response to this need, a study was conducted to evaluate the Perception of Narrative Performance Scale that assesses perceptions of narrative health messages. A sample of 1185 fifth graders in public schools at Phoenix, Arizona completed a questionnaire rating of two videos presenting narrative substance use prevention messages. Confirmatory factor analyses were computed to identify the factor structure of the scale. Consistent with prior studies, results suggest a 3 factor structure for the Perception of Narrative Performance Scale: interest, realism, and identification (with characters). In addition, a path analysis was performed to test the predictive power of the scale. The analysis shows that the scale proves useful in predicting intent to use substances. Finally, practical implications and limitations are discussed.

Keywords: narrative, media engagement, narrative messages, substance use prevention, factor analysis

Health messages presented in various media (e.g., TV, film, videotape, or DVD) often utilize dramatic features and techniques that have been proven effective in health prevention and promotion (Hecht, Corman, & Miller-Rassulo, 1993; Kalichman, 1996; Polansky, Buki, Horan, Ceperich, & Burrow, 1999; Rye, 1998; Warren, Hecht, Wagstaff, Elek, Ndiaye, Dustman, et al. 2006). Narrative health messages have been often examined in the context of popular entertainment media (Moyer-Guse, 2008) such as situation comedy, Friends, daytime TV drama, The Bold & Beautiful (Kennedy, O’Leary, Beck, Pollard, & Simpson, 2004), and primetime TV drama ER (Brodie, Foehr, Rideout, Baer, Miller, Flournoy, et al. 2001). In addition, narrative health messages have been investigated in a serial radio drama (Singhal & Rogers, 2001; Smith, Downs, & Witte, 2007) in the countries where TV was not popular medium.

One of the keys to narrative messages is their ability to engage the audience (Green, Strange, & Brock, 2002). In contrast to more didactic forms, narrative messages have the potential to involve the audiences on a deeper level. As a result, engagement with narrative messages is an important concept for understanding message effects (Roser, 1990; Slater & Rounter, 2002; Vorderer, 1993; Wirth, 2006). While a number of studies have examined engagement as an underlying factor of persuasion (Hinyard & Kreuter, 2006; Slater & Rounter, 2002), less is known about reactions specifically to performances of narrative or education messages in health prevention and promotion. To evaluate the persuasive power of these messages, this study examines a multidimensional measure that assesses audience perceptions of engagement with narrative health messages, called the “Perception of Narrative Performance Scale.”

Perceptions of engagement lie at the heart an of audiences’ experience of narrative messages delivered through performances (Bundy, 2003; Green, Strange, & Brock, 2002). The theory of narrative engagement (Berleant, 1993; Miller, Hecht & Stiff, 1998) is deeply rooted in the philosophy of art and aesthetics. The theory views art objects/events, such as theatrical performances, as differing along a continuum of “cognitive and emotional distance” ranging from engaged to disengaged (Ben Chaim, 1984; Miller et al., 1998). On this continuum the more cognitive/emotional distance, the more disengaged from the art object or event while a minimum level of distance can be conceptualized as high engagement.

Engagement has the potential to provide guidance in constructing narrative messages as well as explaining the persuasive power of audience perception as a mediator of the effects of narrative performances. Performed narratives differ in form from other health messages such as those providing statistical or informative evidence (Slater, Rounter, & Long, 2006). Drama and film theorists suggest that plot (patterns of events structuring a story) and character make up two major components that lead audience members to engage in a narrative (story) and develop a relationship (i.e. closeness) with characters and situations in the performance (Kinkaid, 2002; Miller, Alberts, Hecht, Krizek, & Trost, 2000; Moyer-Guse, 2008; Smith, Downs, & Witte, 2007). In turn, a sense of engagement with plot and character motivates an audience toward insight and action (Klaver, 1995; Miller, Alberts et al., 2000; Schrank & Engels, 1981; Slater & Rounter, 2002; Slater, Rounter, & Long, 2006). Thus, perceptions of engagement may play a prominent role in audience members’ behavioral change. In other words, these perceptions can act as significant mediators in explaining the nature of message effects and, conversely, play an important role in designing messages to enhance impact. As a result, campaign researchers and practitioners have shown interest in measuring the perception of engagement in order to develop more effective theories and increase the efficacy of health prevention messages (Green 2004; Green & Brock, 2000).

Conceptualizing Engagement

Although a great deal of research has conceptualized media engagement as personal relevance or issue involvement, those conceptualizations may not be appropriate for explaining audience responses to narrative and entertainment messages (Hinyard & Kreuter, 2006; Slater & Rounter, 2002). Alternate conceptualizations have sparked controversy in terms of defining the construct of engagement as interest, narrative involvement, and identification or in investigating it more in terms of transportation – that is, the capacity for one to be absorbed into a narrative (Green & Brock, 2000).

We argue that there are important distinctions between the conceptualizations, although both are highly related to narrative message perception. In one sense, a level of engagement can be established from merely attending to a message while in other senses it involves becoming highly moved by it both cognitively and emotionally. At the most basic level, one considers engagement the intensity of attention to the message (Green, Strange & Brock, 2002). We label this attention intensity “interest.” At the next level, one can assess involvement with the message. This denotes a type of association and connection with the message that goes beyond merely attending to it (Green & Brock, 2000; Green, 2006). Involvement, itself, targets the two components which come from drama and narrative theory – plot and characters. That is, the viewer can be involved in the story, a construct we call “realism”, and/or the characters, a construct we call “identification”. Finally, beyond interest and involvement, a viewer can experience a deeper level of immersion into a message which involves a change in cognitive or emotional state; this deeper level can be called transportation (Green & Brock, 2000.).

For the purposes of the Perception of Narrative Performance scale we conceptualize engagement as involving the first two levels – interest and involvement. In this definition engagement consists of attending to and becoming connected with a narrative message. We stop at this level because we believe that transportation denotes a fundamentally different experience and its role in persuasion is distinct from the other two levels. In contrast to our conceptualization of engagement, the transportation-imagery model proposes that transportation involves a deep level of immersion with or absorption into the narrative (Green & Brock, 2000; Green, 2006). It denotes a type of cognitive and/or emotional movement or a shift in consciousness that is reflected in Green and Brock’s (2000) transportation scale items that read, “I could picture myself in the scene of the events described in the narrative” and “I was emotionally involved in the narrative while reading it.” To us, engagement means that a person attends to and forms an attachment with the message while transportation means that the person’s state of mind has changed, either projected into the message or to some other “place” or state. We see engagement as a necessary part of transportation because a message recipient must attend and connect to the message to be moved by it. This, too, is reflected in the Green and Brock (2000) measurement study in which they validate their transportation measure by linking it to realism and identification, two aspects of engagement in our conceptualization. Based on these conceptualizations we offer the following definitions:

  • An interested message recipient is paying attention to the message.

  • An involved message recipient sees themselves as connecting to the plot (realism) and characters (identification) in the message.

  • A transported message recipient sees themselves as paying attention and actively immersed or absorbed into the message or moved to some other reality or mind state.

Not only is transportation conceptually distinct, Green, Garst, and Brock (2004) suggest that transportation should not prove as crucial to educational health messages since audience may not reach deeper level of narrative involvement due to their short-length and overtly persuasive nature. As a result, we focused on interest, realism, and identification.

Miller and colleagues (1998) defined interest as the degree of attention to a performative message. Interest has been considered an important dimension of engagement by numerous researchers (Berleant, 1993; Roser, 1990; Winstron and Cupchik, 1992). When audience members are interested they are more likely to be engaged with the performance whereas they are less likely to feel engaged by a performance that bores them.

The second dimension, realism, was defined as the audience’s experience of the story or plot of a narrative health message as authentic. Audience members tend to disengage from a health message perceived as unrealistic and perceptions of realism and believability act as important indicators of engagement with a narrative message (Slater, Rounter & Long, 2006; Wilson & Busselle, 2004). Realism may increase the impact of health messages by enabling the mimicry of experiences and providing more authentic models that better lead to vicarious learning (see social cognitive theory, Bandura, 2002; Beltramini, 1988; Miller et al., 1998).

Finally, Miller et al. define the third dimension of identification as the degree to which audience members feel similar to a character presented in narrative health message (Cohen, 2001; Liebes & Katz, 1990; Slater & Rounter, 2002; Slater, Rounter & Long, 2006). Feeling “at one” with both the characters and the action presented in the narrative can lead to the unity of experience indicated by engagement (Miller et al., 1998). In the literature on performance, audience members’ identification appears to act as a prerequisite to gaining insight into characters (Schrank & Engels, 1981). Furthermore, identification, in the form of perceived similarity between audience members and characters, plays a significant role in the influence of a narrative message on the audiences’ attitudes or behaviors (Bandura, 2002; Slater & Rounter, 2002).

Measuring Engagement

Audience perceptions of engagement have been measured using various techniques (e.g., Cage & Rosenfiled, 1989; Perse & DeArmond, 1980). To date, however, the field lacks scales for measuring the multidimensional nature of engagement with narrative performances of health messages as conceptualized here and thus a void exists in evaluating this message form. Based upon the theoretical conceptualizations of engagement described above, Miller and her colleagues (1998) proposed a new scale consisting of three dimensions of audience perception of engagement: interest, realism, and identification. These three dimensions reflect audience members’ perception of the degree of attention to narrative, performative messages presented in media. Each of these dimensions is seen as part of the global perception of engagement; although they do not represent an exhaustive list of all of the relevant constructs.

Current Study

The current study uses two drug prevention video messages to test the multi-dimensional nature of the “Perception of Narrative Performance Scale” (see Appendix). This scale was initially developed and pilot-tested by Miller and colleagues (1998) with an older, high school-age, and predominantly White sample. We extend their findings by examining the psychometric properties of the scale with a younger and predominantly Mexican-heritage sample. The contrasting age and ethnic make-up of the current sample allows us to provide stronger evidence for the generalizability and external validity of the scale. The sample characteristics also provide an important extension in application, since 1) Mexican-heritage youth are part of the fastest growing minority in the U.S. (Novello, Wise & Klineman, 1991) and 2) preadolescent youth are targeted more often than older adolescents by prevention messages in the belief that it is better to prevent a risky behavior than stop it once youths have engaged in it (Ringwalt, Hecht, & Hopfer, in press). Finally, this paper provides evidence for the predictive validity of the scale, specifically in terms of its ability to predict substance use expectancies and intentions within the context of a larger intervention study.

The study considers two hypotheses:

  • Hypothesis 1: The perception of narrative performance scale has a three factor structure (interest, realism, and identification).

  • Hypothesis 2: Perceived interest in and realism of the examined video messages will predict less positive expectancies of alcohol and cigarettes as well as lower intentions to use those substances.

Methods

Study Design

Data were collected in 16 public elementary schools in Phoenix, Arizona as a part of a larger intervention study of the 12-lesson keepin’ it REAL substance use narrative prevention curriculum. This study utilizes two sets of data to: 1) perform factor analyses to confirm the three dimension structure of the Perception of Narrative Performance Scale, and 2) assess the predictive validity of two factors in the scale, interest and realism. Both sources of data were provided by students in the treatment condition (i.e. they received the substance use prevention curriculum) within the larger, randomized clinical trial.

The confirmatory factor analysis data (video analysis data) came from 5th grade student responses to the Perception of Narrative Performance Scale, which the students completed after viewing two separate anti-drug videos in class. Of the students participating in the curriculum (N = 1185), 1092 youth responded to the perception scale for the video from a lesson on refusal skills (4th in the sequence) and 1043 youth responded to the scale for the video from a lesson on avoiding substance use offers (6th in the sequence).

The predictive validity data came from cross-sectional questionnaire responses from 1151 individuals from the same cohort of students. These questionnaires were the second post-test evaluating the curriculum. The predictive validity analyses asked students to indicate what they “thought about the keepin’ it REAL videos that were part of the lessons,” rather than having students focus on one video in particular. The design resulting in many of the same students participating in all three of these dataset (two video analysis, one predictive validity) due to the practical limitations of school-based research. This design limits the type of analyses that can be conducted; a limitation that will be discussed in later sections.

Target Messages

The narrative-based curriculum, which includes 5 video-based lessons, has proven efficacious in reducing substance use in a previous randomized clinical trial (Hecht, Graham, & Elek, 2006). Participation in the video component of the curriculum demonstrated an independent effect on substance use (Warren et al., 2006). The introductory videotape tells a story which overviews the curriculum development and the main elements of the lessons. The remaining four, strategy-specific videos emerged from narrative interviews with adolescents. The videos use narrative form (story) to present a substance offer along with a relationally-competent way of refusing or resisting the offer and set these narratives in the contexts in which youth substance use naturally occurs (Hecht & Miller-Day, in press; Warren et al., 2006). The four videos cover the four strategies of refuse (simple no), explain (no with an explanation), avoid (avoid the offer) and leave (leave the situation) (Hecht & Miller-Day, in press).

The confirmatory factor analyses involved the ratings of only two of the strategy-specific videos due to practical limitations involving the available time for evaluation purposes. The selected videos focused on the refuse (“Breaking Bad Habits”; 6 minutes and 22 seconds long) and avoid (“Fiesta”; 6 minutes and 13 seconds long) strategies. The plot of “Breakin’ Bad Habits” involves a break dance contest in which a youth’s team members offer marijuana. The youth simply says no, and that ends the offer sequence. In “Fiesta” a Latina is at a party with friends and avoids a young man who is drinking by engaging her friends in conversation and turning away from his approach. Using the two videos allowed us to evaluate the psychometric properties of the scale across two different messages, providing a more rigorous test than that provided by examination of a single message.

Participants

No identifiers or demographic information could be collected on the data from surveys used in the confirmatory factor analyses due to IRB restrictions at the university collecting the data as well as time and school constraints. As a result, data for the predictive validity sample includes many of the same students. The study students in the intervention schools reported a mean age of 10.4 years (SE = 0.60 years). Females were 49% of the participants. As an indicator of low socioeconomic status, 91% of the students reported taking part in the “free or reduced price lunch program.” Approximately 78% of the students self-identified as Mexican, Mexican-American, or Latino; 7% self-identified as Black; 6% self-identified as White; 3% self-identified as Native American; less and 1% self-identified as Asian American; and 5% self-identified with some other racial/ethnic group. This demographic breakdown matches that from the wave of data collected a few months prior to the video rating data collection sheets, lending confidence to their accuracy in describing all samples used in this study.

Measures

Theorizing and Miller and colleagues’ (1998) exploratory study suggested that their original 14-item scale consists of three discrete dimensions matching their theoretical domains: interest, realism, and identification. The scale used in this study (see Appendix) was modified from the original to better enable the younger sample of pre-adolescents to respond by shortening it to 9 items and using less complex and more readable language. The shortened length is more appropriate for a younger population and also makes it more practical for use in evaluating actual intervention campaigns where instrument length is an important practical consideration. The resulting measure consists of three dimensions with three items for each dimension, with each item rated on a 4-point scale. Before analysis, the item “I was bored during the video” was reverse coded, so that all items measured positive perceptions.

Due to length restrictions in the overall instrument, the predictive validity dataset included only the six items that represented two dimensions of the scale, interest and realism. The analyses tested whether the scores on the two factors (interest and realism) correlated with two substance-related outcomes, positive substance-use expectancies and intent to use. The analyses focus on alcohol- and tobacco-related items as youth of this age report the highest rates of use of those substances (Johnston, O’Malley, Bachman, & Schulenberg, 2009).

Positive expectancies

Positive expectancies represent participants’ perceptions of the positive consequences of alcohol and cigarette consumption. For alcohol, students responded to the item, “drinking alcohol makes parties more fun,” (from Hansen and Graham; 1991) while for cigarettes, students responded to the item “smoking cigarettes makes people less nervous.” (Miller-Day & Kam, in press). Both items were scored using a 4-point agree-disagree scale (1 = “strongly agree”, 2 = “agree”, 3 = “disagree”, 4 = “strongly disagree”) with the scores reverse-coded to represent positive expectancies.

Intent to use alcohol and cigarettes

We measured intentions as a proxy for use given the young age of the sample and the resulting restrictive range of actual use. The two separate items were created for the larger intervention study and consisted of “If you have a chance this weekend, would you use alcohol” and “If you have a chance this weekend, would you use cigarettes.” Students responded to both items on a four-point scale indicating the degree of likelihood of use (1 = “definitely yes,” 2 = “yes,” 3 = “no,” and 4 = “definitely no”). This item was reverse-coded for the ease of interpretation.

Data Analysis Plan

To assess the factor structure of the scale, the analyses used confirmatory factor analysis (CFA). For the two video lessons (i.e., refuse and avoid videos), two individual CFAs were implemented using LISREL 8.80 (Joreskog & Sorbom, 1998). The full information maximum likelihood (FIML) method was used to address missing data (Graham, Cumsille, & Elek-Fisk, 2003). FIML has been shown to yield unbiased estimates when data are missing at random (MAR) and operates robustly even when the MAR assumption is not perfectly met (Enders, 2001; Schafer & Graham, 2002). Since χ2 is influenced by sample size, in particular given this study’s large sample, we focused on three indices of practical fit for model evaluation: the root mean square error of approximation (RMSEA), the squared root mean residual (SRMR) and the comparative fit index (CFI) (Browne & Cudeck, 1993). For both RMSEA and SRMR, smaller values indicate better fit. Following the convention of Hu and Bentler (1999), RMSEA < .06 and SRMR < .08 were considered favorable. As for the CFI, values closer to 1 are preferred and, in particular, values of .95 or above were considered satisfactory (Kline, 2005; Hu & Bentler, 1999). Finally, given that the scale was applied to a different population (Mexican-heritage) with a shortened, modified version for younger kids, the analyses estimated and compared single-factor, two-factor, and three-factor models for each of the lessons.

For the predictive validity analyses, a path analysis was implemented using LISREL 8.80 in order to test the second hypothesis that the perception of engagement would be negatively associated with positive expectancies about alcohol/cigarette use and intentions to use those substances. The FIML method was used to accommodate missing data (Graham, Cumsille, & Elek-Fisk, 2003). Two dimensions of the scale, interest and realism were entered as exogenous variables, while positive expectancies and intentions were used as endogenous variables. To control for the effects of demographic characteristics, gender and age were included in the model as exogenous variables, from which pathways were drawn to the endogenous variables.

Results

Descriptives and Correlations between Items

Table 1 presents the means and related standard deviations for each of the items, along with correlations between the items for the ratings of the two separate videos (confirmatory factor analysis sample). Internal consistency (reliability) appeared of some concern, with Cronbach’s alpha coefficients for the dimensions when rating the Refuse video at .63 for interest, .52 for realism, and .88 for identification. The coefficients for the Avoid video were .75 for interest, .66 for realism, and .93 for identification. These reliability coefficients for interest and realism are less than ideal and suggest further scale development. Inspection of inter-item correlations revealed that the bivariate correlations were all positive and of magnitude roughly ranging from 0.10 to 0.80.

Table 1.

Summary Statistics and Bivariate Correlation Matrices for Refuse and Avoid Video Lessons

Refuse Video (N = 1092) M SD 1 2 3 4 5 6 7 8 9
1. The video was interesting 3.51 0.66 —
2. It was easy to pay attention to the story 3.41 0.72 .34** —
3. I was bored during the video 3.51 0.80 .39** .35** —
4. The video looked real to me 3.45 0.82 .27** .19** .26** —
5. The story was very believable 3.37 0.81 .29** .25** .32** .33** —
6. I could see kids getting into a situation like that 3.25 0.90 .12** .13** .12** .17** .25** —
7. My best friends 2.35 1.30 .11** .04 .07* .08* .06 .11** —
8. My other friends 2.30 1.17 .13** .08* .10** .10** .09** .11** .68** —
9. Me 2.38 1.34 .15** .07* .13** .08* .10** .13** .75** .66** —
Avoid Video (N = 1043) M SD 1 2 3 4 5 6 7 8 9
1. The video was interesting 3.40 0.69 —
2. It was easy to pay attention to the story 3.32 0.76 .62** —
3. I was bored during the video 3.30 0.94 .48** .42** —
4. The video looked real to me 3.30 0.80 .40** .38** .32** —
5. The story was very believable 3.34 0.69 .51** .50** .38** .54** —
6. I could see kids getting into a situation like that 3.14 0.92 .33** .32** .21** .28** .35** —
7. My best friends 2.47 1.68 .17** .19** .17** .18** .16** .14** —
8. My other friends 2.45 1.44 .14** .19** .15** .16** .18** .16** .79** —
9. Me 2.47 1.83 .18** .22** .18** .20** .18** .18** .84** .77** —
Open in a new tab
*

p < .05;

**

p < .01

Dimensionality of Perception of Narrative Performance Scale

To test hypothesis one that predicted a three-factor solution for the scale, three-factor oblique CFA models were run for both video lessons (i.e., refuse and avoid videos). All the model-fit indices revealed that the three-factor models appeared to be a good fit to the data [Refuse video: χ2(24) = 36.64, RMSEA = .022, SRMR = .026, CFI = .996; Avoid video: χ2(24) = 45.32, RMSEA = .029, SRMR = .024, CFI = .996]. The inter-factor correlations between interest and realism were high (.77 for refuse video; .78 for avoid video), whereas the correlations between identification and the other two factors (i.e., interest and realism) were low, ranging from .17 to .26. To deal with the possibility that the strong correlation between the two factors could mean that they were not statistically distinct, a two-factor model merging the interest and realism factors was compared to the three-factor model. Although the two-factor model produced a reasonably good fit, tests of the χ2 difference between the two-and three factor models showed that the three-factor models significantly improved fit for ratings of both lessons: χ2DIFF (2) = 47.69 and 92.68, for the Refuse and Avoid videos, respectively, both p < .001. Consistent with the difference tests, other practical fit indices (i.e., RMSEA, SRMR, and CFI) indicated that the three-factor models fit better than the two-factor models for both videos.

Our conceptualization of engagement suggests realism and identification are the components of involvement, whereas interest just represents attentiveness to a narrative message. For the reason we tested another two-factor model merging realism and identification, and then compared to the three-factor model. Based on practical fit-indices, the model fit was not acceptable and it was much worse than the three-factor model. χ2 difference test also indicated that the three-factor model was better fit to the data than the two-factor model for Refuse and Avoid lessons: χ2DIFF (2) = 471.70 and 1063.67. Both are statistically significant (p < .0001).

Practical fit indices consistently suggested that single-factor models fit poorly for both lessons. The χ2 difference tests also supported the three-factor structure over a one-factor structure as the differences for both lessons were statistically significant: Refuse video: χ2DIFF (3) = 1066.24 and Avoid video: χ2DIFF (3) = 2279.37, both p < .0001. On the basis of the series of model comparisons, the first hypothesis was supported (see Table 2).

Table 2.

Model-Fit Comparison of One-, Two- and Three-Factor Models of the Perception of Narrative Performance Scale

Fit Indices Drug Prevention Video Messages
Refuse Video Avoid Video
1-Factor 2-Factor I 2-Factor II 3-Factor 1-Factor 2-Factor I 2-Factor II 3-Factor
χ2 (df) 1104.9 (27)** 84.3 (26)** 508.3 (26)** 36.6 (24)* 2325.4 (27)** 138.0 (26)** 1109.0 (26)** 45.3 (24)*
CFI 0.737 0.980 0.835 0.996 0.562 0.979 0.794 0.996
SRMR 0.145 0.035 0.116 0.026 0.175 0.033 0.179 0.024
RMSEA 0.191 0.045 0.130 0.022 0.286 0.064 0.200 0.029
Open in a new tab

Note. 2-Factor I indicates a two-factor model merging interest and realism and 2-Factor II indicates a two-factor model merging realism and identification dimensions.

*

p < .05;

**

p < .01

Item-level inspection of the three-factor models revealed that all the factor coefficients were statistically significant and reasonably high, ranging from .52 to .87 for the refuse video and .58 to .93 for avoid video. We found that the item: “I could see kids getting into a situation like that” was not a relatively strong indicator of the realism factor (.35 for refuse; .46 for avoid video). However we did not drop the item because the loadings appeared reasonable and without cross-loading. Table 3 displays factor loadings across the two video lessons; the factor structures clearly resemble each other. Multi-group analyses comparing perceptions of the two videos, which are often used to test the statistical significance for the equality of factor loadings across groups, were inappropriate here. The samples overlap greatly since the ratings were collected from mostly the same students in the same classes, which resulted in nonindependence of the groups.

Table 3.

Factor Structure of Perception of Narrative Performance Scale

Item Descriptions Refuse Video Avoid Video
Interest Realism Identification Interest Realism Identification
The video was interesting .63 (.034) .81 (.023)
It’s easy to pay attention to the story .54 (.036) .77 (.029)
I was bored during the video .64 (.038) .58 (.035)
The video looked real to me .52 (.041) .66 (.031)
The story was believable .65 (.047) .80 (.030)
I could see kids getting into a situation like that .35 (.036) .46 (.041)
My best friends .87 (.018) .93 (.012)
My other friends .78 (.019) .85 (.015)
Me .86 (.018) .91 (.013)
α for scale .63 .52 .88 .75 .66 .92
Open in a new tab

Note. Factor coefficients in the table are standardized and the numbers in the parentheses are their standard errors.

Predictive Power of Perception of Narrative Performance Scale

Using data from the predictive validity sample, the path model yielded a good fit [χ2 (22) = 66.24; CFI = .988; SRMR=.035; RMSEA = .042 (90% CI = .030 – .054)]. While some pathways between interest, realism, and the substance use measures did not reach statistical significance, as expected, interest and realism were negatively associated with positive expectancies and intentions to use substances (see Figure 1). The interest dimension had a significant negative association with positive expectancies regarding cigarette use (unstandardized β = −.21, SE = .10, p < .05). Realism was significantly inversely associated with positive expectancies regarding alcohol use (unstandardized β = −.19, SE = .08, p < .05) and intentions to use alcohol (unstandardized β = −.12, SE = .06, p < .05). Overall, these findings support the second hypothesis positing that perceptions of interest in and the reality of a narrative performance are inversely related to positive expectancies and intentions to use alcohol and cigarettes. They also support the ability of the scale dimensions to validly predict proposed outcomes of narrative health education messages.

Figure 1.

Figure 1

Open in a new tab

Path Model Assessing Concurrent Validity (N = 1,151)

Note. Path coefficients in the figure are completely standardized. All the significant pathways are highlighted by boldface and marked by asterisk. The analyses control for demographics (i.e., sex and age) but the figure does not show them for reasons of clarity.

Discussion

The purpose of this study was to continue the development of the Perception of Narrative Performance Scale to assess the perception of aesthetically-presented, visual media-based narrative substance use prevention messages. Consistent with current theorizing and the suggestions of Miller and her colleagues (1998), the confirmatory factor analyses of the items in the present study supported the fit of a three-factor structure of audience perception of engagement consisting of interest, realism, and identification. In addition, the predictive validity path analysis revealed that engagement in substance use resistance stories predicted reduced positive expectancies and intentions to use substances.

There are several limitations in this study. First it was impossible to match the CFA data for all three factors with substance-related outcomes used for the predictive validity. There were no unique identifies in the video analysis data and the predictive validity data set assessed the overall video perceptions after watching all five drug prevention videos, instead of measuring the narrative perceptions for measuring each individual video message. As a result, we were not able to tie the predictive validity analysis outcomes to particular video messages used in the CFA. In addition the predictive validity analysis employed cross-sectional data and thus causality cannot be established. The next step is to assess the role of involvement with narrative health messages in a longitudinal manner.

As noted earlier the predictive validity analysis only included the two dimensions of our scale, interest and realism. According to social cognitive theory (Bandura, 2002) and extended ELM (Slater & Rounter, 2002), identification plays a prominent role in enhancing the effectiveness of narrative media messages. Future study is needed that includes identification to test the fullest impact of narrative health messages as conceptualized in this study.

The Perception of Narrative Performance Scale has the potential to figure prominently in health prevention research and practice by facilitating the design and evaluation of health messages. The result of this study suggest that the three aspects of engagement (i.e., interest, realism, and identification) identified in this study should be considered when designing health performance messages. These factors should be targeted in initial message development and subsequently pilot tested using the instrument and refined in formative research. In addition, the scale should prove useful in evaluating narrative health messages; effective narrative health messages are expected to be interesting and realistic, and encourage identification. Both practices will influence the way campaign developers and practitioners identify engaging and effective message features and enhance the ability of health education messages to influence desired outcomes.

The scale also allows us to advance health message design by assessing optimal levels of engagement. Communication theorists differ on whether or not to aim for maximum engagement. Some argue that the greater the engagement of audiences with entertainment media messages, the more likely the audiences are to be influenced by those messages (Boal, 1979; Green & Brock, 2000; Miller, Alberts et al. 2000; Slater & Rouner, 2002; Smith et al. 2007). Others argue that while engagement is necessary to facilitate audience identification with the characters portrayed, a certain amount of disengagement also is necessary to distinguish fiction from reality (Artaud, 1970; Grotowski, 1968). For instance, Miller et al. (1998) argue that optimal impact will be achieved by messages with which people are engaged but not overly so, in that very high levels of enjoyment may detract from the message itself. Thus, a balance may be needed in the level of engagement so that a heightened level of engagement will not interfere with narrative processing (Green, Garst, & Brock, 2004; Miller et al. 1998). In other words, engagement and outcomes should form a curvilinear relationship. Until now, the lack of an adequate measure of engagement made it impossible to test these propositions. The Perception of Narrative Performance scale should facilitate future studies to examine the factors that facilitate to as well as interfere with narrative message processing in various visual media contexts.

The scale also will allow health researchers and practitioners to advance health communication theory by testing for mediators of narrative effects and adjusting messages accordingly. Using the current scale, future researchers are encouraged to determine if one or more of the factors is particularly influential to mediate narrative message exposure and behavioral change. Thus, identification of these three mediators may prove crucial for evaluating message strategies and specifying effects. The scale also should promote greater understanding of how narrative anti-drug prevention messages function. Rather than merely comparing them as a category to other message forms (e.g., narrative versus statistical form) or focusing on the level of overall engagement with stories or events, we now can specify how narrative health messages vary in quality and type of engagement. Used in this way the scale has the potential to advance our understanding of how narrative health messages work.

Finally, theory and previous research suggest ways to extend the conceptualization and measurement of narrative performance. A measurement study employing multi-dimensional factors of both measures of engagement (interest, realism, and identification) and transportation could help clarify some of related conceptual issues discussed earlier in this paper. A conceptual model was developed that separates transportation and engagement; a distinction that has yet to be empirically tests. Moreover, the current make-up of the Perception of Narrative Performance scale emphasizes cognitive aspects of engagement more so than emotional elements. In recent years, drama theorists have paid more attention to emotional engagement, referred as the feeling of emotional arousal evoked by audio-visual stimuli (Boostrin, 1990; Kinkaid, 2002). For instance, El-Nasr (2007) explored several types of emotional engagement, such as feelings of empathy and discovery which may evoke anticipation within an interactive narrative experience. Future research should strive to incorporate items into the scale which focus on emotional components of engagement. In addition, norms of scores on the scales have yet to be developed across a range of narrative health messages. Regardless, we believe that the Perception of Narrative Performance Scale provides a promising measure of perceptions of videotaped or live performances of narrative health messages that should contribute to theory and practice.

Acknowledgments

This research was supported by Grant Number (RO1 DA005629) from the National Institute on Drug Abuse to The Pennsylvania State University (Grant Recipient), Michael Hecht, Principal Investigator. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health. The authors express appreciation to Dr. Andrew Hayes and the anonymous reviewers for their valuable comments.

APPENDIX. PERCEPTION OF NARRATIVE PERFORMANCE SCALE

INTEREST and REALISM were measured on the following scale:

Please tell us what you thought about the keepin’ it REAL videos that were part of the lessons.

1 2 3 4
Strongly Disagree Disagree Agree Strongly Agree
Open in a new tab

INTEREST

  1. The video was interesting.

  2. It’s easy to pay attention to the story.

  3. I was bored during the video.

REALISM

  1. The video looked real to me.

  2. The story was very believable.

  3. I could see kids getting into a situation like that.

IDENTIFICATION was measured on the following scale:

Please rate how much the main character(s) in the video is like each of the following people. Use the main character who is most like the people in each of the questions.

1 2 3 4
Not At All Like Very Little Like Somewhat Like A Lot Like
Open in a new tab

IDENTIFICATION

  1. My best friends.

  2. My other friends.

  3. Me.

Contributor Information

Jeong Kyu Lee, ClearWay Minnesota.

Michael L. Hecht, The Pennsylvania State University

Michelle Miller-Day, The Pennsylvania State University.

Elvira Elek, RTI International.

References

  1. Atraud A. Victor Corti (Trans) London: Calder and Boyars; 1970. Theater and it’s double. [Google Scholar]
  2. Bandura A. Social cognitive theory of mass communication. Media Psychology. 2002;3:265–299. [Google Scholar]
  3. Ben Chaim D. Distance in the theater: The aesthetics of audience response. Ann Arbor, MI: UMI Research Press; 1984. [Google Scholar]
  4. Beltramini RF. Perceived believability of warning label information presented in cigarette advertising. Journal of Advertising. 1988;17:26–32. [Google Scholar]
  5. Berleant A. Art and engagement. Philadelphia, PA: Temple University Press; 1993. [Google Scholar]
  6. Boal A. Theatre of the oppressed. New York, NY: Urizen Books; 1979. [Google Scholar]
  7. Boorstin J. Making movies work: Thinking like a filmmaker. Los Angeles, CA: Silman-James Press; 1990. [Google Scholar]
  8. Brodie M, Foehr U, Rideout V, Baer N, Miller C, Flournoy R, et al. Communicating health information through the entertainment media: A study of the television drama ER lends support to the notion that Americans pick up information while being entertained. Health Affairs. 2001;20:192–199. doi: 10.1377/hlthaff.20.1.192. [DOI] [PubMed] [Google Scholar]
  9. Browne MW, Cudeck R. Alternative ways of assessing model fit. In: Bollen KA, Long JS, editors. Testing structural equation models. Thousand Oak, CA: Sage Publication; 1993. [Google Scholar]
  10. Bundy P. Aesthetic engagement in the drama process. Research in Drama Education. 2003;8(2):171–181. [Google Scholar]
  11. Cage T, Rosenfield L. Ekphrastic poetry in performance: An examination of audience perceptions of the relationship between poetry and painting. Text and Performance Quarterly. 1989;9:199–206. [Google Scholar]
  12. Cohen J. Defining identification: A theoretical look at the identification of audiences with media characters. Mass Communication and Society. 2001;4(3):245–264. [Google Scholar]
  13. Enders CK, Bandalos DL. The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling. 2001;8:430–457. [PubMed] [Google Scholar]
  14. El-Nasr MS. Interaction, narrative, and drama: Creating an adaptive interactive narrative using performance arts theories. Interaction Studies. 2007;8(2):209–240. [Google Scholar]
  15. Graham JW, Cumsille PE, Elek-Fisk E. Methods of handling missing data. In: Schinka JA, Velicer WF, editors. Research methods in psychology. New York: John Wiley & Sons; 2003. [Google Scholar]
  16. Green MC. Transportation into narrative worlds: The role of prior knowledge and perceived realism. Discourse Processes. 2004;38:247–266. [Google Scholar]
  17. Green MC. Narratives and cancer communication. Journal of Communication. 2006;56:163–183. [Google Scholar]
  18. Green MC, Brock TC. The role of transportation in the persuasiveness of public narratives. Journal of Personality and Social Psychology. 2000;79:701–721. doi: 10.1037//0022-3514.79.5.701. [DOI] [PubMed] [Google Scholar]
  19. Green MC, Brock TC, Kaufman GF. The role of transportation into narrative worlds. Communication Theory. 2004;14(4):311–327. [Google Scholar]
  20. Green MC, Garst J, Brock TC. The power of fiction: Determinants and boundaries. In: Shrum LJ, editor. The psychology of entertainment media: Blurring the lines between entertainment and persuasion. Mahwah, NJ: Lawrence Erlbaum; 2004. [Google Scholar]
  21. Green MC, Strange JJ, Brock TC. Narrative impact: Social and cognitive foundations. Mahwah, NJ: Lawrence Erlbaum; 2002. [Google Scholar]
  22. Growtowski J. Towards a poor theatre. New York, NY: Simon & Shuster; 1968. [Google Scholar]
  23. Hansen WB, Graham JW. Preventing alcohol, marijuana, and cigarette use among adolescents: One year results of the adolescent alcohol prevention trial. Preventive Medicine. 1991;20:414–430. doi: 10.1016/0091-7435(91)90039-7. [DOI] [PubMed] [Google Scholar]
  24. Hecht ML, Corman SR, Miller-Rassulo M. An evaluation of the drug resistance project: A comparison of film versus live performance media. Health Communication. 1993;5(2):75–88. [Google Scholar]
  25. Hecht ML, Graham JW, Elek E. The Drug Resistance Strategies Intervention: Program Effects on Substance Use. Health Communication. 2006;20:267–276. doi: 10.1207/s15327027hc2003_6. [DOI] [PubMed] [Google Scholar]
  26. Hecht ML, Miller-Day M. The Drug Resistance Strategies Project: A communication approach to preventing adolescent drug use. In: Frey L, Cissna K, editors. Handbook of Applied Communication. London: Taylor & Francis; (in press) [Google Scholar]
  27. Hinyard LJ, Kreuter MW. Using narrative communication as a tool for health behavior change: A conceptual, theoretical, and empirical overview. Health Education & Behavior. 2006;34:777–792. doi: 10.1177/1090198106291963. [DOI] [PubMed] [Google Scholar]
  28. Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling. 1999;6:1–55. [Google Scholar]
  29. Johnston LD, O’Malley PM, Bachman JG, Schulenberg JE. NIH Publication No. 09–7401. Bethesda, MD: National Institute on Drug Abuse; 2009. Monitoring the future national results on adolescent drug use: Overview of key findings, 2008. [Google Scholar]
  30. Joreskog KG, Sorbom D. LISREL 8: Structural equation modeling with SIMPLIS command language. Lincolnwood, IL: Scientific Software International, Inc; 1998. [Google Scholar]
  31. Kalichman SC. HIV-AIDS prevention videotapes: A review of empirical findings. Journal of Primary Prevention. 1996;17:259–279. doi: 10.1007/BF02248795. [DOI] [PubMed] [Google Scholar]
  32. Kennedy MG, O’Leavy A, Beck V, Pollard WE, Simpson P. Increases in call to the CDC national STD and AIDS hotline following AIDS-related episodes in soap opera. Journal of Communication. 2004;54:287–301. [Google Scholar]
  33. Klaver E. Spectational theory in the age of media culture. New Theater Quarterly. 1995;44:309–322. [Google Scholar]
  34. Kline RB. Principles and practice of structural equation modeling. New York: Guilford Press; 2005. [Google Scholar]
  35. Kincaid DL. Drama, emotion, and cultural convergence. Communication Theory. 2002;12:136–152. [Google Scholar]
  36. Liebes T, Katz E. The export of meaning: Cross-cultural readings of Dallas. New York, NY: Oxford University Press; 1990. [Google Scholar]
  37. Miller MA, Alberts JK, Hecht ML, Krizek RL, Trost M. Adolescent relationships and drug abuse. New York: Erlbaum Publications; 2000. [Google Scholar]
  38. Miller M, Hecht M, Stiff J. An exploratory measurement of Engagement with live and film media. Journal of the Illinois Speech and Theatre Association. 1998;49:69–97. [Google Scholar]
  39. Miller-Day MA, Kam JA. More than just openness: Developing and validating a measure of targeted parent-child communication about alcohol. Health Communication. doi: 10.1080/10410231003698952. (in press) [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Moyer-Guse E. Toward a theory of entertainment persuasion: Explaining the persuasive effects of entertainment-education messages. Communication Theory. 2008;18:407–425. [Google Scholar]
  41. Novello AC, Wise PH, Kleinman DV. Hispanic health: time for data, time for action. Journal of the American Medical Association. 1991;265:253–255. [PubMed] [Google Scholar]
  42. Perse J, DeArmond W. Critical judgments of performed texts. Communication Quarterly. 1980;28:57–64. [Google Scholar]
  43. Polansky JM, Buki LP, Horan JJ, Ceperich SD, Burrows DD. The effectiveness of substance abuse prevention videotapes with Mexican American adolescents. Hispanic Journal of Behavioral Science. 1999;21(2):186–198. [Google Scholar]
  44. Ringwalt C, Hecht ML, Hopfer S. Drug prevention in elementary schools: An introduction to the special issue. Journal of Drug Education. 39(4) doi: 10.2190/DE.40.1.a. (in press) [DOI] [PubMed] [Google Scholar]
  45. Roser C. Involvement, attention, and perceptions of message relevance in the response to persuasive appeals. Communication Research. 1990;17:571–600. [Google Scholar]
  46. Rye BJ. Impact of an AIDS prevention video on AIDS-related perceptions. Canadian Journal of Human Sexuality. 1998;7:19–30. [Google Scholar]
  47. Schafer JL, Graham JW. Missing data: Our view of the state of the art. Psychological Methods. 2002;7:147–177. [PubMed] [Google Scholar]
  48. Schrank F, Engels D. Bibliotherapy as a counseling adjunct: Research findings. The Personnel and Guidance Journal. 1981;50:143–147. [Google Scholar]
  49. Singhal A, Roger EM. The entertainment-education strategy in communication campaigns. In: Rice RE, Atkin CK, editors. Narrative impact: Social and cognitive foundations. Mahwah, NJ: Lawrence Erlbaum; 2001. pp. 157–182. [Google Scholar]
  50. Slater MD, Rounter D. Entertainment-Education and elaboration likelihood: Understanding the processing of narrative persuasion. Communication Theory. 2002;12(2):173–191. [Google Scholar]
  51. Slater MD, Rounter D, Long M. Television dramas and support for controversial public policies: Effects and mechanisms. Journal of Communication. 2006;56:235–252. [Google Scholar]
  52. Smith RA, Downs E, Witte K. Drama theory and entertainment education: Exploring the effects of a radio drama on behavioral intentions to limit HIV transmission in Ethiopia. Communication Monographs. 2007;74(2):133–153. [Google Scholar]
  53. Vorderer P. Audience involvement and program loyalty. Poetics. 1993;22:89–98. [Google Scholar]
  54. Warren JR, Hecht ML, Wagstaff DA, Elek E, Ndiave K, Dustman P, Marsiglia FF. Communicating prevention, The effects of the keepin’ it REAL classroom videotapes and televised PSA on middle school students’ substance use. Journal of Applied Communication Research. 2006;34(2):207–227. [Google Scholar]
  55. Winstron AS, Cupchik GC. The evaluation of high art and popular art. Visual Arts Research. 1992;18:1–14. [Google Scholar]
  56. Wirth W. Involvement. In: Bryant J, Vorderer P, editors. Psychology of entertainment. Mahwah, NJ: Lawrence Erlbaum; 2006. pp. 199–212. [Google Scholar]
  57. Wilson B, Busselle R. Transportation into the narratives and perceptions of media realism. Paper to be presented at the Annual Conference of the International Communication Association; May; New Orleans, LA. 2004. [Google Scholar]

RESOURCES