Feasibility of an Online and Mobile Videogame Curriculum for Teaching Children Safe and Healthy Cellphone and Internet Behaviors

Abstract

Objective:

Increased prevalence and penetration of cellphone and mobile Internet use have raised significant concerns about children's health and safety by offering new spaces for cyberbullying, harassment, and sexual misconduct. “Cyberhero Mobile Safety” is a videogame-based education program designed using tenets of the capacity model with the goal of instilling the knowledge and skills necessary to safely and productively navigate the mobile online environment. This study evaluates its usability, appeal, and perceived impact and usefulness.

Materials and Methods:

Six educational videogames were part of a program delivered to 3rd–6th grade students (n=108) across six public schools in Upstate New York. Videogame play was electronically captured to evaluate usability. Likeability, acceptability, and perceived usefulness of videogame content were evaluated through postgame questionnaires.

Results:

Videogame usability criteria were achieved on 82.7 percent of the students' gameplays. On a scale from 1 (low) to 5 (high), mean ratings were 4.09 (standard deviation [SD]=1.28) for likeability, 3.54 (SD=1.61) for acceptability, and 4.16 (SD=1.33) for perceived message usefulness.

Conclusions:

The “Cyberhero Mobile Safety” program is a feasible and potentially effective platform for delivering information about safe and healthy cellphone and Internet use to children. Results support the use of the capacity model to design educational videogames because games that aligned with theory principles were reported as having the most impact and being the most useful at shifting children's online behaviors. Future research should directly test the individual components of the capacity model to inform educational game design.

Introduction

The impact of Internet use on children's health is a growing concern because over the past decade 30 percent of youth have experienced some form of cyberbullying,^1–3 “sexting” has been observed among children as young as 10 years old,^4,5 and one-third of youth reported an online harassment within the last year.⁶ These increasing negative experiences have been attributed to the ways youth are using the Internet and highlight the reality that children are not solely victims in media ecologies, but can also be participants or perpetrators.^7,8 It follows, then, that early interventions that enhance skills related to safe, healthy, positive, effective, and conscientious online behaviors can have an impact. Improving components of digital citizenship will not only make the digital environment more civil but could also reduce the likelihood that young people fall victim to major online threats later in life.

Cybersafety programs have been developed to respond to these concerns, but many of these programs emphasize risk protection measures over developing children's knowledge and skills as active, ethical, and critical participants online.⁹ Additionally, warning children about the dangers of media use may increase their attraction to negative media content.¹⁰ Encouraging digital well-being requires going beyond warning against online dangers and requires promoting etiquette, literacy, and security to empower children to become responsible digital citizens.⁹

Finally, available programs may not capture the attention of this generation's youth^11,12 or keep up with current changes in Web-based and mobile environments,¹³ and limited research has evaluated the usefulness of cybersafety programs at engaging and educating children.¹⁴ With gaming as the most popular online activity for children 6–11 years of age,^15,16 videogames may be an effective platform through which to engage them. Considerable evidence has demonstrated that videogames can facilitate learning and serve as effective teaching tools¹⁷ in areas such as school achievement¹⁸ or cognitive abilities.¹⁹ In addition, integrating educational videogames into the classroom has been shown to improve classroom dynamics and students' motivation to learn,²⁰ with success being attributed in part to player satisfaction and game usability.^17,21,22

The “Cyberhero Mobile Safety” program capitalizes on this generation's affinity for the gaming medium by offering six videogames that teach children about the concepts of digital well-being and citizenship.^15,23 The objective of this study was to evaluate the feasibility of “Cyberhero Mobile Safety” among a sample of children in the 3rd–6th grades. We assessed the usability, acceptability, and likeability of games within the program, as well as students' perception of their usefulness. Our goal was to examine the program as a whole and to explore the characteristics of specific games to determine their success at engaging students.

Materials and Methods

The program

The design of the “Cyberhero Mobile Safety” program was guided by a literature review that identified six principal core competencies that addressed the primary cellphone-related safety risks affecting children today²⁴:

• 1. Identity/Reputation Monitoring: Revealing or misusing personal information prevention

• 2. Relationship Management: Bullying and sexual harassment prevention

• 3. Multitasking: Distraction hazard prevention

• 4. Domains of Use: Disengagement prevention

• 5. Responsibility: Illegal download prevention

• 6. Maximizing the Positives: Constructive use promotion

These six educational concepts focused on social responsibility and citizenship rather than on specific dangers to address the dynamic nature of risk and protective factors and to provide participants with the competencies necessary to be ethical, critical participants online.^9,25 Whenever possible, forbidding language and terms like “sexting” were avoided, and instead appealing replacement behaviors were promoted to limit negative responses to the lessons and increase potential effectiveness.^25–27 Following this general approach of promoting citizenship and positive behaviors through skill building, the six core competencies were developed into the iKeepSafe mobile educational matrix,²⁸ a curriculum guide used to design the “Cyberhero Mobile Safety” games (concepts of the matrix relevant to the games are presented in summary form in Table 1).

Table 1.

Description of the Key Concepts and Definitions for the Six “Cyberhero Mobile Safety” Games

Game	Gameplay	Key concepts	Intended behavioral outcomes	Health issues addressed	Usability definition
“Brain Drain”	Players must navigate to avoid mobile phone distractions in order to finish a school paper.	• Multitasking with a mobile phone reduces focus and efficiency for both tasks.	• Focus on one activity at a time. • When on the phone, be only on the phone; when doing an activity, do only that activity.	• Support efficiency and effectiveness. • Prevent distraction hazards.	Obtained a passing grade
“Digiwoog”	A long-form narrative game. The player must complete a series of challenges in order to save the planet.	• Mobile phones are tools that can be used for many things. • Don't share your password with anyone. • Download legally.	• Use mobile phones for what they do best in the appropriate circumstances (like using a map to find your way).	• Support use as resources. • Prevent identity theft and illegal downloads.	Accomplished the mission
“Public Garden”	Players must use rocks to block the path of weeds and prevent them from spreading everywhere—which is what happens with messages they post online.	• The Internet is public space. • When images and information are posted or shared electronically, they are nearly impossible to keep private.	• Consider all potential audiences before posting images and information. • Post images and information that you'd like to share with a wide audience.	• Support reputation management. • Prevent sexting and victimization.	Mastered the final puzzle
“The Spy Who Texted Me”	Players send a spy on a secret mission and choose how to communicate different phases of the mission most effectively (e.g., via text, phone call, in person).	• Different kinds of communication are ideal for different circumstances.	• Choose the most effective mode of communication for the situation—texting, calling, or talking face to face. • Use mobile phones when they are the best choice.	• Support optimal communication. • Prevent miscommunication and misinformation.	Completed the storyline
“Tech Zombies”	To save Woogis from a zombie infection, which spreads when they interrupt in-person interaction to use mobile phones, players sprinkle them with the “cure”—which makes them put their phone away.	• Using a mobile phone while talking to someone in person gets in the way of your conversation and makes it seem like you aren't present.	• When having an in-person conversation, focus on that—save the phone for later.	• Support in-person relationships. • Prevent disengagement and isolation.	Reached the final day
“Woogi Who”	Players have to choose which picture of a friend they can post online by asking their friend deductive questions about what's okay to share.	• When you take pictures or video of others, you are responsible for making sure that picture is used and shared properly. • Gain the permission of the person whose picture you are taking.	• Ensure that both parties are aware that pictures are being taken and that both parties can say no. • Be clear with others about how you'd like them to use your picture.	• Support positive relationship building. • Prevent cyberbullying and harassment.	Correctly solved the puzzle

“Cyberhero Mobile Safety” is hosted and available for free on Woogi World™ (www.woogiworld.com), a virtual educational community for K–6 students that uses the latest gaming and social networking techniques to encourage peer-to-peer communities and asynchronous learning.²⁹ One game (“Digiwoog Disaster”) is also available as a free mobile application.

Videogame design

Development of the “Cyberhero Mobile Safety” games was guided by Shalom Fisch's capacity model, which postulates that a child's working memory can process a limited amount of information at one time and that to prevent competition for attention, educational and narrative content must be simple, explicit, and integrated (as opposed to tangential) so that information will be better retained.^30,31 The six games are described below (Fig. 1), and the strategies used in each are summarized in Table 1:

• 1. “Brain Drain.” This game addresses multitasking by teaching players that engaging in only one activity at a time can increase their attention and usefulness. The game uses distractions within gameplay that parallel those that occur in real life (e.g., phone ringing, receiving a text message) to show how performance suffers as a direct result of the distraction.

• 2. “DigiWoog Disaster.” This long-form narrative game addresses responsible mobile phone use, such as keeping your password private or purchasing online content to download instead of downloading pirated content. Actions within this game directly parallel the behaviors it encourages. For example, when the player needs the guidebook on how to re-assemble the jetpack, he or she must “buy” it from an online bookstore.

• 3. “Public Garden.” This game focuses on reputation and uses the metaphor that online posts are like “weeds” because they are endlessly replicable and can be shared rapidly. The game teaches players to be conscious about what they share online because both intended and unintended audiences may view the content. The gameplay focuses on preventing weeds from growing in a garden to convey the difficulty of controlling content that is shared online.

• 4. “The Spy Who Texted Me.” This game focuses on relationship management and teaches players to understand which modes of communication (in-person conversation, phone call, or text message) are appropriate for certain situations. Mirroring the kinds of choices players might need to make in real life, players select a method of communication for each scenario and are shown the consequence of their choice within the spy narrative.

• 5. “Tech Zombies.” This game focuses on managing online and offline relationships by encouraging players to turn off cellphones while engaging in face-to-face conversations. The game portrays people who turn into zombies if they are on their cellphone while talking to someone else, and players must address this to advance to the next level.

• 6. “Woogi Who.” This game focuses on respecting others' boundaries and teaches players the importance of requesting permission before sharing personal information about others. To reflect real-life actions, the player is required to “ask” his or her friend a series of questions to determine which picture is okay to “post” online.

FIG. 1.

Overview and illustration of the six “Cyberhero Mobile Safety” games. Color images available online at www.liebertonline.com/g4h

Procedures

Participants were 3rd–6th grade students (n=108) from six public schools in Rochester, NY. “Cyberhero Mobile Safety” was integrated as a standard classroom lesson in the computer lab one or two times per week for 1-hour periods. Each lesson period focused on a single game and its concepts, and lesson plans were provided to teachers to standardize game delivery. Following each game play, participants provided anonymous responses to two opinion questions that included no personal, health, or demographic information (e.g., student's name, age, gender). Because this is a project examining the feasibility of an educational curriculum and only anonymous non–health-related information was collected, Boston Children's Hospital Institutional Review Board classified this evaluation as quality improvement and did not require informed consent. The local school districts and superintendent for each of the six public schools approved the implementation and evaluation of the “Cyberhero Mobile Safety” program, and this evaluation was considered to be a part of the educational curricula.

Measures

Objective measures

Usability was evaluated by measuring the number of gameplays for each game, the proportion of gameplays where game-specific success criteria were achieved (criteria typically represented successful completion of the game objectives [see Table 1]), and the length of time required to successfully complete the game. If players did not accomplish game goals, it was considered an indicator of low usability, and therefore the game would be unable to achieve its educational objectives.

Participant-reported measures

After completing each game, participants were presented with two questions: one game-specific question about the game's goal (e.g., “Did this game teach you how to focus on one thing at a time?”) and one general question that could apply to any game (e.g., “Did you learn something from this game that you can use in your own life?”). The game-specific questions were selected randomly from a set of eight questions designed specifically for that game, whereas the general questions were selected randomly from a set of 12 questions that were used across all games. Both question types were used to assess the following (sample questions are listed in Table 2):

• • Likeability was assessed with questions focused on (1) Appeal and (2) Engagement to determine how participants' responded to each game.

• • Applicability was assessed using questions related to (1) Novelty and (2) Relevance to determine participants' perceptions about the appropriateness and applicability of the games' lesson material. Games that players saw as presenting an educational message that was new to them and pertinent to their own lives were considered highly applicable.

• • Perceived message usefulness was assessed with questions focused on (1) Impact and (2) Usefulness to determine participants' perceptions about whether the game influenced their behaviors.

Table 2.

Evaluation Question Categories and Examples

Category	Subcategory	Definition	Question example
Likeability	Appeal	The extent to which the player enjoyed the game	Did you ever want to quit before the game was over?
	Engagement	The player was engrossed in the game.	Would you want to play this game again?
Applicability	Novelty	The game introduced something new to the player.	Did this game teach you something new about how quickly pictures can spread across the Internet?
	Relevance	The message in the game was applicable to the real-life concerns and behaviors of the player.	Would your friends care if you posted an embarrassing picture of them online?
Perceived message effectiveness	Impact	The players agreed to apply the lessons learned in the game to their lives.	In the future, will you turn off your cellphone when doing homework?
	Effectiveness	The player understood the lesson and felt the game taught it well.	Did this game do a good job of teaching you that it can be rude to text while you are talking to someone?

Concerns have been raised regarding the limited capacity that young children have to understand the typical Likert scale response format.³² In this study, similar to previous studies with children, the typical Likert scale was adapted to the developmental level of children by changing the scale to “Yes” and “No” terms.³³ Following this approach, individual items were all phrased as questions with responses on a 5-point scale ranging from 1 (NO!) to 5 (YES!), with certain items reverse-coded so that higher scores always indicated more positive response to the game. To provide an overall evaluation of “Cyberhero Mobile Safety” and to determine the specific approaches and messages that seemed most effective, mean responses by question subcategories were calculated, and values higher than 4.0 were considered successful. The subcategory values were based on smaller sample sizes.

Results

Usability results for each game are displayed in Table 3. For “Public Garden” and “The Spy Who Texted Me,” every game play met the required criterion for usability. Three other games reached that level for over three-quarters of plays. “DigiWoog Disaster” showed the lowest level of usability, with just over half (58.7 percent) of the participants accomplishing the game's goal.

Table 3.

Usability Scores for the Six Different “Cyberhero Mobile Safety” Games

Game	Number of gameplays	Usability (% of successful gameplays) a	Time (minutes per gameplay) b
“Brain Drain”	71	77.1%	3.4 (0.7)
“DigiWoog Disaster”	104	58.7%	20.8 (7.4)
“Public Garden”	58	100.0%	7.7 (2.9)
“The Spy Who Texted Me”	69	100.0%	7.7 (1.0)
“Tech Zombies”	71	75.8%	5.5 (0.9)
“Woogi Who”	70	86.6%	3.7 (1.1)

^a Usability was measured differently across games as defined in Table 1.

^b Data are mean (standard deviation) values.

Mean responses for participant-reported measures are presented in Table 4. For overall Likeability, the mean rating across games for Appeal was 4.11 (standard deviation [SD]=1.25) and for Engagement was 4.08 (SD=1.32). “Woogi Who,” “The Spy Who Texted Me,” and “DigiWoog Disaster” each surpassed the 4.0 cutoff for Appeal, whereas “Public Garden” scored the lowest (mean=3.33, SD=1.53). For Engagement, all games except for “Public Garden” scored 4.0 or above.

Table 4.

Mean Self-Reported Ratings for the Six Different “Cyberhero Mobile Safety” Games

	Likeability		Applicability		Perceived message effectiveness
Game	Appeal	Engagement	Novelty	Relevance	Impact	Effectiveness
“Brain Drain”	3.86 (1.95)	4.00 (1.12)	4.20 (1.69)	3.06 (1.69)	4.54 (1.12)	4.33 (1.18)
“DigiWoog Disaster”	4.40 (0.89)	4.50 (1.41)	3.46 (1.70)	4.13 (1.31)	4.43 (1.25)	4.27 (0.94)
“Public Garden”	3.33 (1.53)	3.17 (2.04)	2.93 (1.77)	2.80 (1.99)	3.33 (1.76)	4.10 (1.52)
“The Spy Who Texted Me”	4.50 (0.58)	4.11 (1.17)	4.14 (1.03)	3.14 (1.41)	3.55 (1.64)	4.41 (0.94)
“Tech Zombies”	3.50 (1.00)	4.10 (1.37)	3.56 (1.63)	3.88 (1.31)	3.60 (1.81)	4.33 (1.63)
“Woogi Who”	4.80 (0.45)	4.43 (0.79)	3.28 (1.84)	4.00 (1.54)	4.40 (1.18)	4.38 (0.96)
Overall for “Cyberhero Mobile Safety” games	4.11 (1.25)	4.08 (1.32)	3.54 (1.66)	3.54 (1.56)	4.01 (1.51)	4.31 (1.09)

Data are mean (standard deviation) values.

The Applicability measures of Novelty and Relevance scored a mean of 3.54 (SD=1.66) and 3.54 (SD=1.56), respectively, and were the only two measures to not exceed an average score of 4.0 across all games. Topics of mobile distraction and communication covered in “Brain Drain” and “The Spy Who Texted Me” were the only topics to receive above 4.0 for Novelty. In terms of Relevance, scores above 4.0 for “Woogi Who” and “DigiWoog Disaster” suggest that students found the issues related to picture sharing and positive uses of cellphones most applicable to their daily lives. “Public Garden” received the lowest scores for Novelty and Relevance and was the only game to score below 3.0 for any category.

Overall, perceived message usefulness in the games was scored greater than 4.0 across all games, with Impact scoring a mean of 4.01 (SD=1.51) and Usefulness scoring a mean of 4.31 (SD=1.09). For Impact, this finding was driven by high ratings from “Brain Drain,” “DigiWoog Disaster,” and “Woogi Who,” as the other three games received scores lower than 4.0. Scores received for the category of Usefulness were highest compared with all other categories. Each game also received a score above 4.0 (including “Public Garden,” even though it was still rated the lowest).

Discussion

The usability results coupled with the overall likeability reports, which are shown to be prerequisite for its success,^17,21 demonstrate that “Cyberhero Mobile Safety” is a feasible platform for delivering an educational curriculum about cellphone and Internet safety in sample of 3rd–6th grade students.

In order to inform future iterations of “Cyberhero Mobile Safety” and other similar programs, we sought to identify topics relevant to our target population and determine whether the educational content influenced student learning. The main lessons that students reported as relevant to their lives centered on attending to real-life relationships, using cellphones in positive and effective ways, and asking permission when sharing personal pictures. Games that scored high in relevance also tended to be seen as effective and engaging. The two games with the highest relevance scores (“DigiWoog Disaster” and “Woogi Who”) both model positive and responsible use of mobile Internet. Future programs designed to intervene on Internet behaviors may wish to present appealing alternative options to help ensure that their content is perceived as relevant by their young audience and to increase the likelihood of meaningful impact.

According to the capacity model and verified by empirical evidence, factors such as high user interest and explicitness of informational content can direct cognitive resources to the educational programming.³⁰ This theory has been mainly used to design educational television programming³¹; however, findings from our study suggest that this theory may also apply to educational videogame and mobile game programs. Although our aim was to apply tenets of the capacity model universally to the design of each game, subjectively speaking, we were more successful in this goal for some games than others. For example, “Woogi Who” gameplay very closely reflected and reinforced the educational goal (i.e., to win the game, the player had to engage in a simulation of the skill being taught). The distance between narrative and educational content in this game was small, which, according to the capacity model, should have reduced the competition between the two for cognitive resources.³¹ “Public Garden,” on the other hand, used a fairly complex metaphor to convey its message, and the puzzle-based gameplay did not directly reinforce the message. Although blocking weeds with “dirt” to reduce their growth is a metaphor for the spread of information online, this approach may have been confusing or otherwise unappealing to players, and translation of this content may have increased cognitive demand on the player, making fewer resources available for the educational content.³⁴ Across all participant reports, “Woogi Who” scored higher than “Public Garden,” potentially reflecting the more successful use of the capacity model in the former. Other gameplay features, however, may also account for these differences.

Patterns of results for other games may also inform the use of the capacity model in the design of educational videogames. For example, “DigiWoog Disaster” had the lowest usability score but had the second highest ratings in likeability and was perceived by students to have a high degree of impact and to be useful at shifting their online behaviors. “The Spy Who Texted Me” ranked high on usability and likeability but was rated lower than “DigiWoog Disaster” in its ability to influence their behaviors. Usability, it seems, did not guarantee and was not necessary for self-reported influence. The capacity model states that simple programs (ones more likely to score high on our usability measure) are easier for children to cognitively process. However, it may be the case in videogames that when games are easy to master, the interest in the content decreases and reduces information absorption. This pattern is repeated in the games “Brain Drain” and “Tech Zombies”: These games were higher than “DigiWoog Disaster” in usability, yet they were given lower scores within appeal and engagement categories. Future research should explore whether likeability mediates the link between usability and impact or whether each factor independently affects the impact of education information.

Limitations

Our sample was relatively small and from a single geographic region (Rochester, NY), which limits generalizability of the findings. This study primarily relied on self-reported data, and issues such as social desirability may have been enhanced in the classroom setting, where participants may have felt pressured to rate the program more highly because it was implemented by their teachers. To reduce the likelihood of this bias, participants were informed that these games were not part of their educational assessment, responses were voluntary and anonymous, and teachers did not have access to the results. The variability in scores, including means around the midpoint of the response choices, provide at least some evidence that the students did not all feel inclined to rate the games highly. Although in-game measures of usability provided a more objective assessment of gameplay, the cutoff points for usability were somewhat arbitrary. Considering that our goal was to describe student responses, these metrics were chosen as merely comparison points and do not offer absolute indicators of game success or failure. Furthermore, full scales delivered to every student were not feasible because of time constraints. Although this technique limited the number of responses, we feel our results provide a suitable representation of student perceptions. A full evaluation examining the impact of the behavioral effect of these games would necessarily require more thorough measures.

Conclusions

Although additional work is necessary to document the efficacy of these games at shifting actual behaviors, these preliminary findings are promising and provide initial evidence that “Cyberhero Mobile Safety” games are feasible and useful for 3rd–6th grade schoolchildren.

Our results suggest that games that aligned closely with the capacity model were more likeable, explicit, and related to the educational messaging and were also seen as more useable. High likeability and relevance scores suggest that designers of educational gaming programs should frame their content in an approachable, flexible, and solution-based format to engage and influence children's perceptions. A more systematic investigation is warranted to understand the direct or indirect relationships between the tenets of the capacity model and to determine whether certain tenets are more influential than others at driving educational information retention.

Acknowledgments

Funding for this project was provided in part by the Internet Keep Safe Coalition through their corporate sponsors and their grant from the Office of Juvenile Justice and Delinquency Prevention, U.S. Department of Justice (2009-MC-CX-K065). We would like to thank Marsali Hancock and her staff at IKeepSafe, Scott Dow and his staff at WoogiWorld, and the school districts, teachers, and students involved in this project.

Author Disclosure Statement

No competing financial interests exist.

Y.H. drafted the manuscript and conducted the analyses for the current study, L.R. and D.B. were involved in the conception and design of the project, and D.B. is Principal Investigator and oversaw the entire project. All authors critically reviewed and edited the drafts of the manuscript.