Analysis of the Roles of “Serious Games” in Helping Teach Health-Related Knowledge and Skills and in Changing Behavior

Abstract

Researchers are developing sophisticated games specifically targeted to teach health-related knowledge and skills and to change health-related behaviors. Although these interventions, generally called “serious games,” show promise, there has been limited evaluation of their effectiveness. This article offers a broad “consumer guide” for evaluating such health education interventions. Improving the development and evaluation of healthrelated serious games and educating potential purchasers of such products to be knowledgeable, demanding consumers will help move the field of serious games from “looks promising” to determining where such interventions will be effective and where they will not.

For centuries, games have been used to impart knowledge and skills. The potential for using games as teaching tools has increased with the development of computer technologies, which have allowed games to become more complex while making game-based approaches easier to implement. Many games have been developed specifically to teach health-related knowledge and skills and to change health-related behaviors.¹ “Serious games”— those that teach real-word knowledge and skills—have been developed in many health-related areas, including hygiene, public safety, healthy eating, AIDS education, obesity prevention, and diabetes treatment.

Although there has been research and development in the area of serious games, there has been limited evaluation of the effectiveness of such games to attain their goals, i.e., causing the players to increase their knowledge or to change their behavior. The article by Dr. Thompson and colleagues in this issue of Journal of Diabetes Science and Technology provides a clear example of both how a theory based game intervention might be developed in an area of health and how such a game might be evaluated. The article describes the theoretical framework, design, and alpha testing of personal computer-based video games that seek to teach preadolescents knowledge and skills and to change behaviors regarding diet and physical activity. The authors' objective in developing these interventions is to decrease obesity and type 2 diabetes in adolescents using the game via building skills in goal setting and then reviewing their success in meeting their goals, with interactive feedback. The authors note that other game-based interventions have been used to promote youth dietary change, food acceptance, and physical activity. The study included a built-in evaluation component. The developers of these games used focus groups to test basic elements of the game (e.g., characters, story lines), and they reported on the initial testing of key aspects of games, including the setting and review of health-related goals and the usability and comprehensibility of the games. Follow-on work planned by the team will involve quantitative analyses of learning, and eventually of behavioral change.

The Thompson study highlights the importance of getting feedback from the target audience early in the design process and provides an example of best practices in game-based interventions by showing the close integration of game development and evaluation. This integrated framework can be used by both game developers and consumers.

Quick Guide to Evaluating Game-Based Health Interventions

Although most readers of this journal will not design game-based health interventions such as those described in this issue, readers may at some point be in the position to evaluate or use such interventions as part of their professional practice. The better informed users are in selecting or reviewing commercial game offerings, the more game developers will be pushed to improve the quality of these products and to demonstrate their value.

Those who are considering using a “serious game” intervention should focus on both the quality (and outcomes) of the content they teach and the “game” aspects of the application. It is possible to have a game that is theoretically based, strong in pedagogy, and contains a well-designed content but is boring to play and thus ineffective with the target audience. “Time on task” is one of the great general “truisms” of educational interventions: the longer one spends learning, generally, the more one learns. In particular, if the goal is to engage children or teens in a health intervention, then the intervention needs to be appealing as a game. However, on the other end of the continuum, there are games that are engaging, i.e., have great graphics, motivating point systems, and levels of play, but teach little relevant content or, worse yet, may engagingly teach the wrong content or inappropriate health behaviors. The point is that both the “game” and the “health content” have to be well done to achieve the goals of the intervention designers.

To help shape the market for such interventions, a person seeking a game-based, health-related intervention for children should ask the following kinds of questions of the developers/purveyors of the intervention:

• What theoretical model underlies the design? (Theory based?)

• What are the approvals or rationales for the specific health knowledge/skills/behaviors taught? (Approved best practices?)

• How much time does the average child spend in relevant educational activities in a session, and how many sessions do they play? (Time on task?)

• What are the measurable skills and outcomes in the game, and how do they map onto the child's real world? (How assessed?)

• What are the quantitative results that suggest that this is effective? Are they users' “reactions,” “pre- and post-test learning outcomes,” or “evidence of actual behavioral changes?²” (Actual results?)

If an intervention passes muster on these basic questions, the next test is to see how the game is actually used by target users (which will almost certainly be different from how it was just demonstrated). That is, outside of the aspects of the game demonstrated by the developers, does the game provide long-term engagement for the target audience? Adults are notoriously bad at estimating what will be engaging to children or teens. However, usability testing with even just a few members of the target audience can usually provide some useful feedback on the intervention. Using a demonstration copy of the intervention, many of today's children can quickly assess whether the gaming experience is “lame,” “hollow,” or genuinely fun and challenging. In short, kids need to review applications aimed at kids.

General Need for Better Design and Evaluation of Serious Game-Based Interventions

Designing effective game-based health interventions will continue to be a blend of science and art: a successful development team needs an integrated set of professional skills. Foremost, the team must include strong expertise in what succeeds in educating or changing behavior in the real world and in clinical interventions, which is hopefully supported by an underlying theoretical framework. This expert domain knowledge must be complemented by expertise in game design, which is an art in itself. Academics and clinicians should not believe that they, as sophisticated consumers of games themselves or parents of children who play games, know how to blend the elements of game mechanics, reward systems, levels of play, graphics, animation, sound, and graphical design. Regardless of the merits of the underlying intervention, if the target audience is not engaged by the game, it will not be played and learning will not take place.

Finally, the development team must also include expertise in evaluation. This aspect of the team should be integrated from the start of the project for two reasons. First, rigorous assessment requires measurable outcomes. Defining the learning or behavioral change goals for the intervention may shape the design of the game or curriculum presented during the course of the game. Second, the assessment may require actions within the game itself to be measured, as opposed to a pre- or post-test paradigm. The assessment might be based on the number of times an error is made or according to changes in patterns of choices in the game (e.g., selection of different food types appropriate to children with diabetes).

Designing strong evaluations, gathering good data, and analyzing and communicating results are all key to moving the field of serious games from “looks promising” to determining where such interventions will be effective in areas of health education and where they will not. Carrying out such evaluations requires significant investments from funders to finance appropriate evaluations and from researchers to include evaluation expertise as part of their development teams. Without results from serious evaluations, of serious games like those pursued by Dr. Thompson and colleagues, game-based health interventions will continue to “show promise,” but still remain on the fringes of health education practices.