Creating a Novel Video Vignette Stroke Preparedness Outcome Measure using a Community Based Participatory Approach

Abstract

Evaluating the efficacy of behavioral interventions for rare outcomes is a challenge. One such topic is stroke preparedness, defined as inteventions to increase stroke symptom recognition and behavioral intent to call 911. Current stroke preparedness intermediate outcome measures are centered on written vignettes or open ended questions and have been shown to poorly reflect actual behavior. Given that stroke identification and action requires aural and visual processing, video vignettes may improve upon current measures. This article discusses an approach for creating a novel stroke preparedness video vignette intermediate outcome measure within a community based participatory research partnership. A total of 20 video vignettes were filmed of which 13 were unambiguous (stroke or not stroke) as determined by stroke experts and had test discrimination among community participants. Acceptable reliability, high satisfaction and cultural relevance were found among the 14 community respondents. A community based participatory approach was effective in creating a video vignette intermediate outcome. Future projects should consider obtaining expert and community feedback prior to filming all the video vignettes to improve the proportion of vignettes that are usable. While content validity and preliminary reliability were established, future studies are needed to confirm the reliability and establish construct validity.

Acute stroke treatments are underutilized. About 3% of acute ischemic stroke patients receive tissue plasminogen activator (tPA), which is a time-limited, clot buster drug proven to reduce post-stroke disability (Adeoye, Hornung, Khatri, & Kleindorfer, 2011). The overwhelming reason that stroke patients do not receive tPA is their late arrival to the hospital, outside of the tPA treatment window. (D. Kleindorfer et al., 2004). Calling 911 is associated with a decrease in time from stroke symptom onset to arrival to the emergency department (Rosamond, Gorton, Hinn, Hohenhaus, & Morris, 1998). Thus, stroke preparedness interventions, defined as inteventions that aim to increase stroke symptom recognition and increase behavioral intent to call 911, are needed.

Despite nearly 800,000 strokes annually in the US, the probability that one person will witness a stroke is low, making outcome assessment of stroke preparedness interventions a challenge (Luepker et al., 2000; Morgenstern et al., 2002). For example, the REACT study was a mutlilevel intervention to decrease the time from onset of acute myocardial infarction symptoms to hospital arrival. This negative study randomized 20 communities in 10 states and collected data from over 20,000 patients (Luepker et al., 2000). Similair to the REACT study, stroke preparedness intervention trials would need to randomize communities in order to have sufficient sample size to evaluate clinical outcomes resulting in expensive and resource intense trials (Luepker et al., 2000; Morgenstern et al., 2002). We recognize that clinical outcomes such as tPA utilization are preferable to intermediate end points (Atkins, 2004). Yet, especially in the initial phases of development and testing of stroke preparedness interventions, a compelling role for an intermediate end point exists. An intermediate end point allows researchers to test several interventions before progressing to phase 3 testing and facilitates intervention testing for selected populations, including high stroke risk groups such as African Americans (AAs) (Kissela et al., 2012).

Currently, stroke preparedness research is limited by few and faulty intermediate outcome measures. These measures use open ended questions or written vignette assessment measures and do not accurately reflect behavior (Dawn Kleindorfer et al., 2008; Morgenstern et al., 2007; Williams & Noble, 2008). Scores of these written assessments yield much higher behavioral intent than behavior observed among acute stroke patients/witnesses, suggesting that current measurements could be improved (Hsia et al., 2011). Video vignette outcome measures may expand upon the current written measures. Videos allow for both aural and visual processing of situations and reduce the impact of limited literacy (Chan & Schmitt, 1997). An additional strength of video vignettes is that it allows participants to identify their course of action despite having never been in the situation, removing key confounders such as literacy levels and over simplification of situations causing obvious outcomes (as may have been the case in the written vignettes assessment). Researchers who have compared video vignettes to live situations found that behavioral intent in vignette responses closely mirrored actual live behavior, a necessary characteristic of intermediate outcomes (Dalessio, 1994; Fleming & DeMets, 1996; Schulman et al., 1999).

This paper presents a case study of a community based participatory research approach to create a novel video vignette intermediate outcome measure that will facilitate testing and prioritizing of stroke preparedness behavioral interventions. We present the benefits of using a community based participatory approach for video vignette measure development that other researchers may find useful in creating video vignette outcome measures; we also provide suggestions for improvements in our development process.

Methods

This study was conducted within the Preparing to ReAct Immediately to Stroke through Education (PRAISE) partnership in Flint, MI. Over 100,000 people live in Flint, of which 57% are AAs and 37% live below the poverty level ("State and County Quick Facts. United States Census Bureau. 2010"). Once the center of the US automotive industry, Flint’s steady economic decline has resulted in a current unemployment rate of 15.1%. Genesee County, where Flint is the largest city, has one of the highest age-adjusted stroke hospitalization rates and stroke mortality in Michigan (BE Anderson). Given the great burden of stroke within Flint, the community is committed to stroke prevention and preparedness.

PRAISE academic partners include stroke neurologists, health behavior and health education experts from the University of Michigan. PRAISE community partners include Bridges into the Future, a faith based community organization dedicated to improving the health of the Flint community (Skolarus et al., 2011). The goal of PRAISE is to develop and test a faith based stroke preparedness intervention among AA youth and adults in Flint, MI (Skolarus et al., 2013). PRAISE is predicated on the Theory of Planned Behavior developed in 1991 and derived from the Theory of Reasoned Action (Ajzen, 1991). The Theory of Planned Behavior guides all components of this intervention and outcome assessment. This theory maintains that the most immediate and accurate predictor of observable behavior change is an individual’s intention to change behavior (Armitage & Conner, 2001); behavioral intention indicates that the individual is ready to perform the actual behavior. Because we anticipate that participants who score higher in their behavioral intent to call 911 for stroke on the video vignette measure will also be more likely to perform this task in a real life situation, the video vignettes will serve as the outcome measure for the PRAISE intervention.

Creation of PRAISE stroke vignettes scripts

Development of the video vignettes was critical, as we aimed to minimize confounders and maximize our ability to determine behavioral intent. Several methods were used to develop the video vignettes, including pre and post-production focus groups and post-production surveys (see Figure 1). The PRAISE team first developed a list of characteristics important to have represented in each vignette – a range of settings and behaviors fitting to the AA Flint community, a mix of genders and ages, and accurate depictions of appropriate social structures and cultural nuances. Twenty short video vignettes scripts were created across 4 settings (i.e., sets for filming) - a grocery store, living room, restaurant and basketball court. A board certified, fellowship trained stroke neurologist drafted vignettes focused on the most common stroke symptoms, facial droop, arm weakness and speech/language difficulties, which PRAISE participants will learn during the stroke preparedness intervention (D. O. Kleindorfer et al., 2007). Video vignettes emphasize stroke symptoms including facial weakness/dysarthria (n =3), arm weakness (n = 7), speech/language difficulties (n=5), and included 5 non-stroke vignettes. The vignettes were designed to be 30-90 seconds long and include dialogue between 2 actors. Because youth will participate in the PRAISE intervention, and are an important part of stroke preparedness in this community, 2 of the 4 sets included a youth. Each vignette ends with 2 multiple-choice questions that assess: 1) stroke recognition (yes, no, don’t know); and 2) action response for this vignette (call doctor’s office immediately, wait a couple of hours and then decide, call a friend/family member immediately, call 911 immediately) (Billings-Gagliardi & Mazor, 2005). Vignette scripts were reviewed by multiple PRAISE stakeholders. Stroke experts (n=10) including stroke neurologists, general neurologists, emergency department physicians, nurses, nurse practitioners and stroke study coordinators provided the first round of review that resulted in two major changes to vignettes. First, the basketball setting was determined to be too difficult to distinguish stroke symptoms from expected minor injuries during basketball (such as a sprained wrist), and was replaced by an office setting. Second, the number of vignettes with isolated speech/language disturbance was decreased, given concern for potential difficulty among actors trying to mimic speech/language disturbances. Vignette scripts underwent a second round of review by community partners as well as PRAISE community and youth advisory boards. Changes resulting from this review largely affected dialogue and linguistic components of the vignettes, particularly vignettes including adults and youth, as the PRAISE team strove to ensure cultural competence and community appropriateness.

Figure 1.

Overview Development of the Video Vignettes

Filming of PRAISE stroke video vignettes

A local production team was chosen to produce the videos and put out a casting call to local actors. Prior to auditions a stroke neurologist selected youtube video clips of stroke symptoms (including speech/language difficulties, slurred speech and facial droop) and provided these to the production team for use during auditions. The PRAISE team reviewed the audition tapes and selected actors. All filming was performed in Flint with a stroke neurologist, health education expert and community partner present, allowing the PRAISE team to observe and offer suggestions to actors and the production team regarding realistic representation of stroke symptoms during filming and subsequent editing.

Review of filmed PRAISE stroke video vignettes

After filming and editing, 20 video vignettes were shown to 6 stroke experts (3 stroke neurologists, a stroke coordinator and 2 stroke nurses). Vignettes were discarded if experts could not reach consensus about stroke diagnosis. To ensure that community members (beyond PRAISE partners and advisory boards) would find video vignettes acceptable, the PRAISE team solicited greater community feedback by recruiting individuals to participate in a focus group discussion of vignettes at this stage. Many participants are members of Community-Based Organization Partners (CBOP) of Flint, an alliance of community-based organizations established to strengthen the influence of community partners involved in community based research projects in Flint (Griffith et al., 2010). Focus group participants were recruited via word of mouth and invited to attend a discussion session following a CBOP meeting in space donated by the community.

PRAISE academic and community partners conducted the video assessment and focus group. Community participants were consented by one of the PRAISE academic partners. After viewing each of the 20 stroke video vignettes as a group, participants were asked to individually complete stroke recognition and action response questions for each vignette. Participants then completed a survey which included sociodemographics and ratings for: 1) satisfaction; 2) ease of completing vignettes; 3) whether the settings were familiar; 4) whether character interactions were believable; 5) whether social network structures were accurately portrayed; and 6) whether customs portrayed were representative of their culture/community on a 5 point likert scale (1-strongly disagree to 5-strongly agree). These statements were modeled after a culture sensitivity and readability scale for materials for AAs (Guidry, Fagan, & Walker, 1998). After completing the stroke video vignettes and survey, PRAISE researchers and community participants had a 60 minute feedback session centering on their perception of the videos. The focus group session was not recorded, but copious notes were taken during discussion. Focus group participants received a small incentive in appreciation of their time.

Analysis

Descriptive statistics were used to summarize participants’ sociodemographics. Knowledge of stroke warning signs and behavioral intent to call 911 were calculated. Reliability of the novel vignettes was evaluated using Cronbach’s α. Content validity was established during the review of the written and video vignettes by stroke experts. Notes from all PRAISE meetings, meetings with production team and the community viewing were reviewed. Statistical analysis was performed using Stata 11.0 (StataCorp, College Station, Texas).

Results

A total of 20 vignettes were filmed (see Figure 2). Of the 20 videos vignettes, 2 stroke vignettes were discarded after stroke experts did not agree that the symptoms unambiguously represented a stroke. Vignettes were then shown to community participants. A total of 14 CBOP adults and 1 youth participated. The median age of the adults was 60 (IQR 41-63) and 78% were women. Ninety three percent of adult participants had at least some college and 21% had advanced degrees. Two participants had previous medical training as either a certified nursing assistant or a nurse/caregiver. All community participants identified stroke in 4 out of 18 video vignettes and reported they would call 911 in one additional vignettes. Given that the vignettes are intended to stratify participants on their levels of stroke preparedness, these 5 vignettes were discarded. Thus, 13 stroke vignettes remained (8 stroke and 5 non-stroke). The 8 stroke vignettes represented each of the cardinal symptoms of stroke of which 4 of 8 represented various degrees of arm weakness. Acceptable reliability was found among the adult community respondents for recognition of stroke warning signs (Cronbach’s α = 0.74) and behavioral intent to call 911(Cronbach’s α = 0.63). The final selected video vignettes are available on youtube at https://www.youtube.com/channel/UCY0Xd4u40CPQw6LWy72uC-w.

Figure 2.

Selection of video vignettes

Satisfaction with the video vignettes among the community participants was high (table 1). Ninety three percent of the community participants strongly agreed that they were satisfied completing the video vignettes and 71% strongly agreed that the video vignettes were culturally appropriate. Community participants had practical, negative and positive comments. Practical suggestions included numbering each vignette to make completion of the video survey easier. One participant also suggested adding to the survey instructions “assume the symptoms do not go away and ignoring cost, please answer these scenarios.” One participant requested an outdoor set which was lost when the basketball set was cut. Negative comments included the concern that the vignettes focused on only 2 people, which was thought to be unrealistic. Another suggestion was to have the stroke occur while the person is in a group. The office setting portraying a younger AA man reporting to his supervisor who is an older AA man proved controversial. One man reported that “although wishful, is not very representative [referring to both working in an office and the two AA men reporting to each other].” Other participants voiced that although it is rare for AA men in Flint to work in offices rather than factories, this vignette might occur in venues such as churches and schools.

Table 1.

Characteristics of primary sample

Positive comments outweighed negative comments. One woman remarked how “It was good it didn’t look like all the people/scenes were in the ghetto. It’s okay if you are from the ghetto, but not everyone is from the ghetto.” Another remarked how “It was very important that all the actors looked like us.” And another person “Well put together, it made you think on what to do. “ Finally, participants were pleased that the production team, sets and actors were in their community. One woman voiced “It is important to work with the community and helping to turn things around for the better. Unity is the key.”

Discussion

This study highlights the benefits of a community based participatory approach to ensure cultural appropriateness and content validity when creating a novel stroke video vignette intermediate outcome measure. The high satisfaction and cultural relevance scores as well as qualitative data provide evidence of cultural appropriateness. Content validity is supported by video vignette script creation by a fellowship trained, board certified stroke neurologist and extensive review of both scripts and videos by stroke and health behavior/health education experts. Although reliability within the current sample was acceptable, the ability to perform a rigorous psychometric analysis was limited by the small sample size. Repeating this study with a larger, more representative sample should be considered. Future studies should also consider the assessment of construct validity by comparing stroke recognition and behavioral intent to call 911 scores among participants of various levels of stroke knowledge and ultimately comparing to clinical outcomes. Nonetheless, these novel stroke video vignettes represent the first step in the creation of an intermediate outcome measure to evaluate the efficacy of stroke preparedness interventions.

The importance of garnering community feedback when creating videos that represent actual medical scenes is exemplified in this study. For example, community participants noted that the majority of vignettes included only two individuals, one of whom witnesses the onset of the stroke. While this decision was made so as not to distract from the stroke patient, it may make identification of stroke symptoms easier and could have contributed to the 5 discarded videos which lacked discrepancies in stroke recognition and/or action response among participants. In future stroke video vignettes, we recommend adding vignettes in which new stroke symptoms are noticed by the witness after a brief time away from the stroke patient, as well as the inclusion of more than 2 actors per scene. To identify these dynamics and improve quality of each scene, we recommend filming half of the video vignettes and securing feedback from content and community experts prior to filming the remainder of the vignettes. Re-filming some of the vignettes was not an option in this case due to budgetary constraints and thus 7 vignettes were excluded.

The most controversial vignettes were the office set, which illustrates the complexity in developing videos through a community based participatory approach; not everybody has to relate to the videos, but an effective vignette should empower community participants by portraying situations that play on real life experiences of community participants. In the script writing stage, there was mixed review of this set with some participants finding it acceptable whereas others were concerned that it may not be relevant to the Flint community. The PRAISE team decided to alter the scene to take place at a church office or a school office. Despite this change, the office set remained controversial during the community focus group. Original fears that the vignette may not accurately represent the community were again echoed in this viewing. This concern did not diminish the group’s overall positive response to the vignettes, and also serves to illustrate that referring to a single ‘African American perspective’ is impossible; all racial and ethnic communities in the US have wide heterogeneity and stroke preparedness should reflect that.

In conclusion, a community based participatory approach proved paramount in creating this novel video vignette outcome measure. While content validity and preliminary reliability were established, future studies are needed to confirm the reliability and establish construct validity. The video vignettes will be used to test PRAISE, a planned faith-based stroke preparedness intervention among AA youth and adults in Flint, MI, which will further explore the reliability and construct validity of the measure. Obtaining community feedback at every stage of development, and particularly after filming half of the videos, provides the potential to alter the remaining scripts and may result in more video vignettes being used in the final video vignette outcome measure.