The design and implementation of a randomized controlled trial of a risk reduction and human immunodeficiency virus prevention videogame intervention in minority adolescents: PlayForward: Elm City Stories

Abstract

Background

To address the need for risk behavior reduction and HIV prevention interventions that capture adolescents “where they live,” we created a tablet-based videogame to teach skills and knowledge and influence psychosocial antecedents for decreasing risk and preventing HIV infection in minority youth in schools, after-school programs, and summer camps.

Methods

We developed PlayForward: Elm City Stories over a 2-year period, working with researchers, commercial game designers, and staff and teens from community programs. The videogame PlayForward provides an interactive world where players, using an avatar, “travel” through time, facing challenges such as peer pressure to drink alcohol or engage in risky sexual behaviors. Players experience how their choices affect their future and then are able to go back in time and change their choices, creating different outcomes. A randomized controlled trial was designed to evaluate the efficacy of PlayForward. Participants were randomly assigned to play PlayForward or a set of attention/time control games on a tablet at their community-based program. Assessment data were collected during face-to-face study visits and entered into a web-based platform and unique real-time “in-game” PlayForward data were collected as players engaged in the game. The innovative methods of this randomized controlled trial are described. We highlight the logistical issues of conducting a large-scale trial using mobile technology such as the iPad®, and collecting, transferring, and storing large amounts of in-game data. We outline the methods used to analyze the in-game data alone and in conjunction with standardized assessment data to establish correlations between behaviors during gameplay and those reported in real life. We also describe the use of the in-game data as a measure of fidelity to the intervention.

Results

In total, 333 boys and girls, aged 11–14 years, were randomized over a 14-month period: 166 were assigned to play PlayForward and 167 to play the control games. To date (as of 1 March 2016), 18 have withdrawn from the study; the following have completed the protocol-defined assessments: 6 weeks: 271 (83%); 3 months: 269 (84%); 6 months: 254 (79%); 12 months: 259 (82%); and 24 months: is ongoing with 152 having completed out of the 199 participants (76%) who were eligible to date (assessment windows were still open).

Conclusion

Videogames can be developed to address complex behaviors and can be subject to empiric testing using community-based randomized controlled trials. Although mobile technologies pose challenges in their use as interventions and in the collection and storage of data they produce, they provide unique opportunities as new sources of potentially valid data and novel methods to measure the fidelity of digitally delivered behavioral interventions.

Introduction

Minority youth are disproportionately affected by human immunodeficiency virus (HIV) infection.¹ Since younger minorities are at increased risk, it is critical to tailor interventions to help them avoid behaviors that put them at risk for HIV infection. A major challenge in HIV prevention for adolescents is capturing them in their environment – meeting them “where they live.”

The prevalence of videogame playing is increasing with most youth playing videogames daily.² “Serious videogames” (games for a primary purpose other than entertainment)³ can serve as interventions to affect behavior change and increase knowledge in youth.⁴ We strove to harness the potential of mobile videogames to deliver a risk reduction and HIV prevention intervention to adolescents. Evaluating the impact of behavioral interventions has inherent challenges that videogame technology may address. For example, there are concerns about the validity of self-reported data.⁵ Videogame software produces data on activities within the game that may be a proxy for real world actions,⁶ mitigating some of these issues. In addition, delivering behavioral interventions with fidelity, or according to protocol,⁷ can be challenging. Electronic interventions allow for the provision of standardized content, and gameplay or in-game data document every action the player takes in the game, time spent on each discrete action, and their overall performance providing a method of assessing exposure to different intervention components.

We produced an iPad game, PlayForward: Elm City Stories and conducted a randomized controlled trial (RCT) to evaluate its impact on targeted outcomes. By incorporating the core efficacious elements of established prevention interventions into an interactive tablet game, this delivery method may have significant impact on retention in the intervention, its reach through mobile devices, and its ability to promote beneficial and sustainable changes.⁸ This article describes the unique methods employed to conduct an RCT using a serious videogame as a behavioral intervention. It outlines novel approaches for managing large amounts of data produced through videogame software. We describe how in-game data may correlate with traditional assessment data and the role videogames may play in providing information about intervention fidelity. Our goal is to provide insights and guidance to researchers interested in evaluating videogames and digital media as behavioral interventions.

Methods

Overview

The first 2 years of this research were dedicated to the development of the PlayForward intervention through a collaborative and iterative process working with researchers, videogame development teams, adolescents and community programs staff.^9–12 PlayForward, an interactive videogame grounded in a conceptual model of health behavior change, addresses the development of behavioral skills to avoid or reduce risk behaviors including those associated with HIV. It is rooted in social learning theory, self-efficacy, message framing, and delay discounting. The goal of the game was to help adolescents acquire knowledge related to risk, safety, and health, to learn to navigate peer relationships, and to understand the benefits and consequences of their behaviors. The subsequent RCT is evaluating the efficacy of the videogame intervention on preventing or reducing risk behaviors, and increasing knowledge and intentions to engage in healthy behaviors. Participants were assigned to play PlayForward, or attention/control games on an iPad tablet. The primary outcome of the RCT was delay in the initiation of sexual activity. Data collected by study personnel through face-to-face assessments at baseline and follow-up were entered in a secure web-based system. In-game data were collected through the PlayForward’s software. No in-game data were collected from the attention/time control games.

Study setting and participants

A total of 333 adolescents aged 11–14 years from 12 urban after-school, school-based, and summer camp programs, were enrolled in the RCT. The 12 sites provide services for primarily racial/ethnic minority youth and consist of seven in-school after-school programs, four freestanding after-school programs, and one summer camp program. At any given time, 6–28 participants were engaged in the study at a specific site. All study activities, including gameplay and data collection, occurred on-site at the participants’ programs.

Eligibility criteria

Eligible participants were adolescents, aged 11–14 years, English-speaking, who were able to provide assent and parental/guardian consent, and were willing to participate in playing PlayForward or control games for 60–75 minutes twice weekly for 6 weeks.

Study entry and informed assent/consent

Recruitment, enrollment and randomization occurred over a 15-month period. Recruitment processes included distributing flyers to adolescents and their parents at their program and presentations at parent-teacher meetings. Study personnel discussed the study with interested adolescents and their parent/guardian. We obtained written informed assent from participants and written informed consent from parents/guardians. Consent forms in Spanish were provided when appropriate. Participants were provided with an age-appropriate description of the study as “finding out how kids can make choices that are healthier for them.” Parents/guardians were informed that the study focused on promoting healthy behaviors in adolescence and on providing strategies to assist adolescents in making decisions about challenges they face. Both the consent discussion with parents/guardians and the consent form itself were explicit about the game content and assessment questions, including about vaginal, oral, and anal sex and substance use behaviors. If an adolescent or parent/guardian did not wish to participate, they did not have any further involvement in the study. The Yale University School of Medicine Human Investigation Committee approved the research.

Randomization

Eligible participants were assigned to play PlayForward, or attention/control games on an iPad tablet. A single randomization scheme was generated and written in TrialDB, a customizable web-based clinical trials database system.¹³ Randomization was stratified by gender and age group (11–12 and 13–14 years). Both variables have been found to be important predictors of outcomes of interest.^14,15 After reviewing the adolescent’s eligibility, the study personnel accessed the TrialDB system to enter the participant’s data, initiate randomization, and retrieve the assignment. We will perform an intent-to-treat analysis.

Study conditions

The intent was for participants to play their assigned game(s), intervention or control, twice weekly for 60–75 minutes per session for 6 weeks, at their program. Study personnel provided participants snacks and a break halfway through each gameplay session to participants. To minimize contamination, the intervention and control groups sat at different tables or in different sections of a classroom. All participants used a dedicated tablet with headphones throughout the study, helping to make their experience an independent one.

Intervention condition: PlayForward: Elm City Stories

PlayForward is a two-dimensional, role-playing adventure videogame about how choices one makes in life impact both short-term and long-term goals. It allows players to acquire and practice skills to reduce risk behaviors focusing on sexual and substance use risk, with the ultimate goal of HIV prevention. The game involves an interactive world where the player, using an Aspirational Avatar they construct, “travels” through life, facing challenges and making decisions in the context of a series of stories and narratives that bring different risks and benefits. The game comprises five skill-based highly interactive “mini-games,” (Table 1) and 12 challenges or story narratives embedded within the PlayForward game.

Table 1.

Description of mini-games

Me Power (Aspirational Avatar)	In Me Power, the player develops their Aspirational Avatar, which embodies the player’s future goals and aspirations, thus creating a reflective gameplay experience.
Refusal Power	In Refusal Power the player must assess risky situations and consider how best to refuse in ways that are not only protective but also preserve the self within the context of social expectations.
Know Power	In Know Power, the player uses their knowledge about health-risk behaviors to win in a battle of the wits. Know Power provides the player with knowledge related to sex, drugs, and alcohol.
People Sense	In People Sense, the player must navigate peer relationships in their social network. People Sense teaches the player about the association between peers’ behaviors and the influence on their own behaviors.
Priority Sense	In Priority Sense, the player must consider the possible positive and negative outcomes involved in making a decision. Priority Sense is about learning how to prioritize behaviors that maximize long-term benefits.

Each PlayForward participant was assigned a unique gameplay ID number. In general, the participants signed into the games themselves, clicking on their gameplay on the iPad to access their game. Both the gameplay ID and progress were automatically saved by the game.

Control condition: off-the-shelf game(s)

A total of 12 videogames such as Angry Birds, SpellTower, and Subway Surfer, loaded onto each control participant’s iPad, served as an attention/time control. These games were devoid of content relevant to our study goals. We chose an attention/time control to mirror the number of sessions and total length of gameplay in the experimental group.¹⁶ Although we considered a conventional HIV prevention education control condition, we opted for an attention/time control, as our primary goal was to establish the videogame’s efficacy. In addition, there appears to be no “gold standard” for teaching sexual education or HIV prevention in US schools.¹⁷ Therefore using HIV prevention/education materials as the control did not seem consistent with current “treatment as usual.”

Control participants did not receive gameplay IDs but instead were identified in the computer database by a specific code, and their unique iPad assignment was linked to their computer-generated study IDs.

Data sources: collection and management

TrialDB was used to collect, store, and access standardized assessment data for this study.¹³

Standardized assessment data

Outcome measures were collected at baseline, 3 and 6 weeks, and 3, 6 and 12 months, with ongoing follow-up at 24 months. The primary outcome is delay in the initiation of sexual activity^18–20 at the 12-month time-point. We also collect data on secondary outcomes and gameplay experience.

Each assessment time-point had a specified window within which collection of the assessment data would be considered “on time” as indicated: 6-week and 3-month assessments (2 weeks); 6-month assessments (1 month); 12-month assessments (2 months); and 24-month assessments (3 months). Participants received gift cards for completing assessments at each study visits: baseline, 12- and 24-month assessments (US$40), and 3- and 6- week and 6-month assessments (US$20). Creative methods were used to re-engage with participants over time to encourage participation in follow-up assessments: text messaging to communicate with participants and their families; geographically localized teen-friendly locations such as libraries and community centers to meet with groups of participants; pizza and cupcake parties.

It was essential to maintain consistent access to the Internet for the purposes of collecting data from participants while on-site. We used wireless hotspot routers for community-based study sites that had either no or sub-optimal access to the Internet for data entry.

We chose to use face-to-face assessments for a number of reasons. First, we were collecting a considerable amount of data (15 different assessment instruments) in a young teen cohort and wanted to ensure that the assessments were completed. Second, given that some of the assessments were collecting sensitive data (sex- and substance use-related data), we were cognizant of issues around disclosure. Although there are differing views in the literature regarding paper versus computer-based assessments of sensitive data,²¹ research has demonstrated more skipped items and no specific advantage to web-based interviews²² and found that paper versions elicited higher and more accurate rates of disclosure.²³

Assessment data were collected prior to the gameplay session. Schedules were arranged to designate days for assessment collection only, such that gameplay and assessment collection did not occur on the same day.

iPad gameplay data for the PlayForward condition

The PlayForward software collected data on each player’s behaviors and actions during gameplay. These real-time and saved-game collected data present unique challenges and opportunities. There are two types of data the game collected: Player Game State Data, that is the traditional save/load data required so that a player can save their progress and then continue later where they stopped in the game, and Activity Logging Data, real-time data that captures relevant actions that players take during gameplay, and which will be extracted and analyzed at the end of the study. For example, every interaction that the player encountered in the game such as selecting options, entering/exiting a game area, or making a choice, was recorded with a timestamp. The data for each of these interactions were stored in text format log files, which can then be imported into a database or spreadsheet for analysis.

Participants assigned to play PlayForward completed between 1 and 14 hours of the game (total potential gameplay time was 16 hours). Given that gameplay data were recorded and saved cumulatively, there was no situation in which the gameplay data was incomplete. We plan to evaluate how performance in or completion of different parts of the game correlate with outcomes reported on the standardized assessments.⁶

Data collection from the videogame occurred as follows: 1) a gameplay session occurred, in which a player made progress in their game on their dedicated iPad; 2) after a gameplay session was completed, research assistants transported iPads to a “Central Data Repository,” a dedicated secured computer locked in the research office; 3) to transfer the data files, the iPads were individually connected to this Central Data Repository via Apple iTunes® software and an iPad Universal Serial Bus (USB) cable; 4) the Player Game State Data and Activity Logging Data files were transferred to and from the iPads using the File Sharing feature of the Apple iTunes application; 5) the study personnel initiated an upload on each iPad, which accomplished the following: a) Player Game State Data were uploaded from the iPads to the database, with the following caveats: 1) if identical Player Game State Data already existed in the database, the new data were not stored and 2) new Player Game State Data did not overwrite old Player Game State Data, even for the same player; instead, all uploads were saved permanently, with the most recently played Player Game State Data considered current.

The Activity Logging Data were uploaded from the iPads to the database, and then deleted from the iPads. It was important that the upload operation was performed for every iPad before performing a download, to ensure that each iPad received the most recent data.

The data remained in the Central Data Repository until a given player’s 6 weeks of gameplay were completed at which time all of the player’s data were transferred securely to a University-managed shared drive. The Player Game State Data file was a binary file instead of plain text for optimization reasons. The Activity Logging Data file was a plain text file to allow for easier data analysis.

Security measures

Data security was accomplished in several ways. Personal identifying information was never stored on the iPads or in the Central Data Repository, and measures were taken to ensure that no in-game data were put at risk of being distributed to unauthorized parties. iPads were always in the possession of authorized personnel or in the possession of study participants (players) under the supervision of authorized personnel. Similarly, the Central Data Repository was secure. Player Game State Data and Activity Logging Data were stored as unencrypted plain text data, but the data were transferred over secure local networks, and not the Internet. Activity Logging Data were uploaded from the iPads to the database, and then deleted. Data were transferred via USB cables from the secured iPads to a university-secured computer. Only study personnel could access saved game files (the Player Game State Data file).

iPad data security was further ensured by activating the “find-my-iPad” function on each device. In the event that an iPad was lost or stolen, the research team would have been able to remotely erase all data and content on the device. Finally, all iPads were locked in safes customized for storing and charging tablets, located in the locked research office, after every data collection session. This assured that all of the devices were secure and charged for subsequent gameplay sessions.

Quality assurance procedures

After developing the videogame intervention and prior to trial recruitment, the research team underwent intensive training on the methods and procedures of the study. A 6-week pilot of procedures, including onsite procedures, implementation of the videogames on the iPads, and data collection and entry was undertaken with three adolescents at one of the program sites to trouble-shoot and resolve problems and to optimize the quality of the process. These data will not be included in the final analysis dataset. Quality assurance protocols were subsequently established and maintained throughout the trial at each of the study sites.

Sample size

Study power calculations were based on data from published studies^24,25 and reviews^8,26,27 of studies investigating the efficacy of youth HIV prevention interventions. These suggest a small to moderate effect size of such interventions on delaying/preventing the initiation of sexual activity. In one study, 31% in the control group versus 12% in the intervention group had initiated sexual activity at 1 year,¹⁹ and in another study, of those who were abstinent at baseline, 14% in the control versus 4% in the intervention group reported initiating sex at 5 months.²⁴ National data indicated that approximately 7% of youth reported engaging in sexual activity prior to age 13²⁸ and this figure was accounted for in the final sample size. The sample size estimation for this study was based on the comparison of proportions of participants that had not initiated sexual activity by 12 months; a sample size of 330 (165 in each study condition) was estimated to provide a power of 0.80 or greater to detect significant (two tail alpha=0.05) differences of such magnitude.²⁹ The sample will also afford adequate power (>90%) to detect small to moderate effects on secondary outcome measures. NQuery v4.0 was used to estimate sample size.

Planned analyses

The primary outcome is the delay of sexual initiation measured at 12 months post-randomization and constructed as a binary outcome (sex delayed beyond 12 months/sex initiated prior to 12 months). The primary comparison will evaluate the effect of PlayForward compared with the control videogames using logistic regression. The primary analysis will be adjusted for gender and age (the randomization stratification variables) and other a priori determined variables. A discrete time hazard model will also be considered to assess change in the primary outcome over time.

Secondary outcomes include other sexual behaviors,¹⁹ knowledge about HIV/AIDS risk behaviors and transmission,^30,31 level of self-efficacy regarding negotiation around initiation of sexual activity,³² drug and alcohol use behaviors,¹⁹ level of self-efficacy in negotiating situations involving offers of drugs and alcohol,³³ and intentions, attitudes and norms around sex.³² Some assessments were implemented in their original form, others were adapted to be more relevant to the target population of younger adolescents.

Other outcome data collected at each interval included personal limits and risky situations,²⁰ social support from friends and family,²⁰ feedback on gameplay experience (developed by the research team), exposure to other health/sex education programs,²⁰ bullying/teen dating violence,¹⁹ future orientation,³⁴ STD/HIV/pregnancy status (testing and treatment),²⁰ youth violence,³⁵ and school connectedness/experiences.^36,37

Differences in scores in these variables will be compared between the two study groups at the time points (6 weeks, 3, 6, 12 and 24 months) using linear repeated measures mixed models to assess changes in continuous variables (e.g. knowledge scores) over the 12 months. Models will include a fixed effect for intervention group, time as a continuous variable (or categorical as appropriate) representing the time-point of assessment and the interaction between intervention and time to assess difference in slopes by intervention arm. Logistic regression models will be considered to determine which baseline variables are individually associated with improvement comparing the PlayForward group with the control group (constructed as a binary outcome (Yes/No)) in these parameters. Variables that are significant at the p=0.1 level in univariate regression models, will be used in the multivariable logistic regression analysis.

The association of PlayForward gameplay data with self-report assessment data

Using Player Game State Data and the Activity Logging Data, we will examine the paths taken and decisions made, comparing across participants, both within and across study arms. We will also evaluate in-game data and their association with our outcomes by examining correlations between in-game and self-reported data using methods we have developed.⁶ Automatically generated logs of gameplay events will be imported into a database table. Logged events recording the completion of a mini-game level will be used to generate player score reports. Composite score variables will be constructed using in-game data (over the 6-week play period) to assess participant performance on specific mini-games. Regression analysis (with adjustment for appropriate covariates such as age or gender) will be used to evaluate the utility of composite scores to predict (1) scores from post-intervention validated assessments and (2) the primary outcome (delay of sexual initiation). We will compare mean changes in assessment scores between the two study conditions and examine the correlation between such changes and the composite scores and their change (at different time points to baseline).

PlayForward gameplay data as an indicator of intervention exposure and fidelity

We can infer periods of inactivity (e.g. player is thinking or away from the device) from time-stamps in the Activity Logging Data. These data provide the opportunity to identify and catalog areas of exposure to specific components of the intervention, offering a method of assessing fidelity to the PlayForward intervention.

Results

In this RCT, participants are recruited, enrolled, and complete the gameplay intervention and assessments in their programs. A study flow diagram is presented (Figure 1). A total of 333 participants were randomized: 47% girls, mean age was 13 years, and 87% minority teens. The randomization procedure was successful; the two groups were similar in terms of gender (boys and girls), age (11–12 and 13–14 years), and race/ethnicity (Black, Hispanic, and White): 53% of the PlayForward group were boys (88/166) and 47% were girls (78/166); the boy:girl distribution was similar in the control group (53.3% (89/167) versus 46.7% (78/167)). The age distribution within each randomized group was also similar–– 11–12 years old 51.8% (86/166) in the PlayForward group versus 51.4% (86/167) in the control group and 13–14 years: 48.2% (80/166) in the PlayForward group versus 48.5% (81/167) in the control group. Ethnicity distribution (Black/Hispanic/White) was balanced–– PlayForward group: 42.7%/55.9%/10.4% versus control group: 40.1%/55.1%/8.6%. A total of 166 participants were assigned to PlayForward (162 or 98% initiated gameplay) and 167 to the control games (159 or 95% initiated gameplay). Reasons for not initiating gameplay included participants’ inability to participate in the afterschool program due to transportation or medical issues. To date (as of 1 March 2016), 18 have withdrawn with 315 participants in active follow-up. A total of 271 (83%) have completed 6-week assessments; 269 (84%) have completed 3-month assessments; 254 (79%) have completed 6-month assessments; and 259 (82%) have completed 12-month assessments. The 24-month data collection is ongoing with 152 having completed out of the 199 participants (76%) who were eligible to date (assessment windows are still open).

Figure 1.

Study flow diagram (as of 1 March 2016).

Discussion

We describe the components of an evaluation of the PlayForward risk reduction and HIV prevention videogame intervention in the context of a RCT. We outline novel methods created and used for collecting and storing in-game data. We developed unique methods for analyzing these data for elements associated with the standardized assessment data and to document intervention fidelity. The videogame intervention also serves as an assessment tool based on the data the game software produces. These data address some of the challenges presented with self-report data and, with further development of analytic methods, in-game data may represent a valid proxy for self-reported outcomes. Given that a challenge of conventional behavior change interventions is documenting exposure to the intervention, videogame interventions offer a unique strategy for assessing intervention exposure.

One of the other major advantages of a mobile videogame intervention, over conventional prevention strategies, is the capacity to reach a larger adolescent population. The incorporation of the core elements of evidence-based prevention interventions into a mobile interactive game may improve retention and reach, and promote sustained changes.⁸ We have enhanced the game’s accessibility with its subsequent translation to additional platforms such as Mac, Windows and Android tablets.

Although the PlayForward videogame only addresses risk behaviors, including those associated with HIV infection, the power of videogame technologies for addressing myriad health and social issues is becoming increasingly clear. Videogame interventions have efficacy in affecting behaviors related to health promotion and disease management in numerous areas.^4,38–43 Accordingly, the fields of “serious games” and “games for health” are rapidly evolving.^44,45 There is also compelling evidence that individuals who acquire and practice new behavioral skills and behaviors in a virtual environment, are more likely to act in accordance with the new skills.^46,47

There are potential limitations to this work including the use of self-reported data for the primary outcome. Some studies have found that reporting of sensitive behaviors is increased when survey methods such as computer versus face-to-face interviews are used.²¹ However, we believe we have used data collection methods that optimize disclosure and the accuracy of self-reported data, and ensure privacy and confidentiality in this age group.^21,22

The potential for technology-based interventions is great and therefore, methods for their evaluation must be rigorous while maintaining an “out of the box” approach so as not to limit the valuable data produced from these trials. The results of this trial will contribute to the literature on the efficacy of videogame interventions. This article provides unique and critical information regarding the methods to assess the impact of these interventions and highlights the significant potential for these data.