Examining an Online Microbiology Game as an Effective Tool for Teaching the Scientific Process

Abstract

This study investigates the effectiveness of the online Flash game Disease Defenders in producing knowledge gains for concepts related to the scientific process. Disease Defenders was specifically designed to model how the scientific process is central to a variety of disciplines and science careers. An additional question relates to the game’s ability to shift attitudes toward science. Middle school classes from grades six to eight were assigned to the experimental group (n = 489) or control group (n = 367) and asked to participate in a three-session intervention. The sessions involved completing a pretest, a game play session, and taking a posttest. Students in the experimental group played Disease Defenders while students in the control group played an alternative science game. Results showed a significant increase in mean science knowledge scores for all grades in the experimental group, with sixth grade and seventh grade students gaining more knowledge than eighth grade students. Additionally, results showed a significant positive change in science attitudes only among sixth graders, who also rated their satisfaction with the game more favorably than students in higher grades. No differences in mean test scores were found between genders for science knowledge or science attitudes, suggesting that the game is equally effective for males and females.

INTRODUCTION

Teachers at all educational levels use textbooks to support science instruction, but studies have revealed the inadequacy of textbooks in representing the scientific process. A recent study published in the Journal of Microbiology & Biology Education determined that introductory undergraduate biology textbooks under-represent the scientific process, which could be an impediment to students’ understanding of science (9). Similarly, the K–12 classroom oftentimes covers the scientific process and required skills in isolation from content at the beginning of the school year with textbooks reinforcing this separation (26). This is problematic because the teaching of science should mirror the processes used by real scientists, which are not separated from content and involve hands-on investigations. How are students at all educational levels to succeed in understanding the scientific process when they have resources that impede their ability to do so? These findings highlight the need for innovative materials that provide students a stronger conceptual depiction of the scientific process.

Digital games can enable students to experience the scientific process by providing virtual laboratory or field investigations that overcome constraints that teachers face, such as lack of laboratory facilities, laboratory supplies, or long distances from science museums (24). Computer-based science games have previously been demonstrated to increase learning in various science content areas (2, 14, 15, 19–22, 27), suggesting a place for games in supplementing science instruction. In addition, computer-based science instruction (28) and a narrative-based online game (14) have been shown to positively influence attitudes toward science, suggesting that digital games could serve as a teaching tool for mitigating the negative shifts in attitudes toward science observed in middle school and high school students (11, 12).

Given the gender gaps that have been reported in science, technology, engineering, and mathematics (STEM) areas (25), developers of any school-based science education program have to be sensitive to potential gender differences in the program’s efficacy, likeability, and usability. Digital games in particular were, for a long time, considered a male leisure activity. However, recent statistics refute major gender differences regarding the playing of video games: 99% of boys vs. 94% of girls reported playing video games in a recent Pew Report (17), and researchers have found that both genders benefit equally from science-based digital games (1, 7, 13).

Our previous research has demonstrated learning gains in microbiology content among middle and high school students who used MedMyst, an online Flash-based Web adventure series containing virtual lab experiments and arcade-style games set within an engaging mystery story line (http://medmyst.rice.edu/) (20). However, the focus of the three MedMyst missions previously evaluated is on teaching microbiology content and science history, not the scientific process. Also, the prior evaluation did not investigate the ability of MedMyst to influence students’ attitudes toward science.

To explore the potential of an online game to teach the scientific process within the context of an engaging microbiology scenario, the current MedMyst Web adventure series was expanded to include an additional mission called Disease Defenders. The objective of Disease Defenders is to illustrate the scientific process, including constructing hypotheses, defining variables, defining experimental and control groups, testing hypotheses, analyzing data, and drawing conclusions. In addition, Disease Defenders portrays the scientific process as utilized by three different infectious disease specialists: a microbiologist, epidemiologist, and veterinarian. The purpose of including three different infectious disease specialists in Disease Defenders is to show students that there is not a single “scientific method” used by scientists, which is consistent with the goals of the conceptual framework for the new K–12 science education standards (7). We hypothesized Disease Defenders to be effective in producing knowledge gains for concepts related to the scientific process among middle school students and we hypothesized middle school students would report more positive attitudes toward science after exposure to the game.

METHODS

Intervention

With input from middle school science teachers, we created several learning objectives that align to the National Science Education Standards (23), which were then used in the game design process to guide the development of the storyline and game activities (Table 1). An advisory panel of infectious disease experts reviewed the game content for scientific accuracy.

TABLE 1.

Learning objectives, in-game activities, and science knowledge questions for Disease Defenders.

Learning Objective	In-Game Activity	Questionsa
Formulate a hypothesis and identify independent and dependent variables.	Microbiology Path: Perform a simple filtering experiment to attempt to remove the rabies pathogen from a liquid.	1, 4, 10, 13, 14, 16b, 17
Use experimental and control groups to draw conclusions from an experiment.	Veterinarian Path: Examine brain samples using microscopy to determine if a pig died from rabies.	6 b, 7, 11, 15b
Interpret results and draw conclusions about a case-control study.	Epidemiology Path: Conduct a case-control study to determine the source of a rabies outbreak.	2, 3b, 5, 8, 9, 12

^aScience knowledge questions can be found in Appendix 1.

^bIndicates general science knowledge questions.

Disease Defenders begins with a short dialogue explaining that investigating disease outbreaks requires a team of experts. Players are then asked to choose, in any order, three different career “paths” (veterinarian, epidemiologist, and microbiologist), each with a virtual infectious diseases specialist portrayed as a “disease defender.” Guided by the on-screen scientists, in the veterinarian path, players perform a diagnostic procedure; in the epidemiology path, players conduct a case-control study; and in the microbiology path, players perform a traditional laboratory experiment (Fig. 1). By involving students in the process of conducting a scientific investigation from three different career perspectives, they experience how the scientific process is applied in different contexts. All of the components of the scientific process are included in each career path; however, each path emphasizes some components more than others (Table 1). For example, the microbiology career path places emphasis on hypothesis development and identification of independent and dependent variables; however, students are still exposed to hypothesis testing, data analysis, and drawing conclusions from the results of an experiment.

FIGURE 1.

From top to bottom, screen captures from the beginning of the veterinarian, epidemiologist, and microbiologist paths in Disease Defenders showing the problems players are asked to investigate.

In addition to the scientific process, rabies is another common thread that runs through each path. However, the focus is not on teaching about the rabies pathogen. Rather, showing how different scientists apply the scientific process to investigate different aspects of the same infectious disease is used to further facilitate students’ contextual understanding of scientific investigations. Based on the results of prior research in which student engagement was positively correlated with learning outcomes, another reason for choosing a common storyline was to maximize the potential for student interest and engagement (27). Placing the content within a storyline also situates the learning in a context that necessitates problem solving and provides structure to the educational content, both of which have previously been established as important factors in producing effective educational games (3, 5, 10, 29, 30).

Another technique used to support learning is allowing players to interact with virtual scientists, who provide instructions on what to do next, give feedback, and check for understanding, as illustrated in Figure 2.

FIGURE 2.

Screen capture of a check for understanding during the veterinarian path of Disease Defenders. Players are asked to identify and add experimental, positive, and negative control brain samples to microscope slides for staining to determine if a pig died from rabies.

Participants

Approval to conduct the study was obtained from Rice University’s Institutional Review Board. An online solicitation was sent out to middle school science teachers who had attended various MedMyst professional development workshops. An incentive of $150 for organizing the field test was offered to teachers. To participate in the field test, science teachers were required to meet the following inclusion criteria: permission from the school principal, access to a computer lab for three class periods, guarantee to provide at least 50 students, and collection of informed consent forms signed by both parents and students.

The final sample included 17 teachers from 15 different public schools across 11 US states. This comprised 856 students (51% female) with 98 of the students in sixth grade (11.4%), 461 in seventh grade (53.9%), and 297 in eighth grade (34.7%). Of the students involved in the study, 72% identified themselves as Caucasian, 12.7% Hispanic/Latino, 8.1% Asian/Pacific Islander, 2.9% African-American, 0.9% Native American, and 3.4% Other. To validate the representation of socio-economic status diversity, students in the sample were assigned to one of three economic groups as operationalized by free/reduced lunch percentages of the total school population, ranging from low disadvantage (less than 29% of students receive free/reduced lunch), to middle disadvantage (29–59% of students receive free/reduced lunch), and high disadvantage (more than 59% of students receive free/reduced lunch). The final sample contained 19.1% (n = 163) low-disadvantage students, 50.1% (n = 429) middle-disadvantage students, and 30.8% (n = 264) high-disadvantage students.

Procedure

The 17 participating teachers were assigned to either the experimental group or control group based on the grade level they teach in order to ensure equal representation of sixth, seventh, and eighth graders between groups. All students who provided parental consent participated in three sessions in the school computer lab. During Session One (approximately 30 minutes), students in both groups were asked to provide their assent, were asked for an identification number assigned by the teacher so students’ identities could not be matched to data, were asked basic demographic information (e.g., grade, gender, ethnicity), and responded to 17 multiple-choice knowledge questions and science-related attitudinal scales. During Session Two, scheduled a minimum of three days after Session One, students in the experimental group played Disease Defenders (approximately 30 minutes total). Students assigned to the control group played a different science game from the MedMyst series, which is comparable in length, but does not focus on the scientific process. Use of a control group in pretest/posttest designs permits understanding whether effects of learning are a function of the intervention or threats to internal validity (i.e., testing effects, history, maturation, or game-play processes). After an additional three-day delay, students in both groups completed the posttest in Session Three (approximately 30 minutes), which contained the same knowledge test and the same science-related attitudinal scales that were in the pretest. In addition, participants were asked to rate the usability and their satisfaction with the game. The minimum three-day time delay between game play and posttesting was purposely planned to test the persistent effects of the game beyond the immediacy of the intervention.

Measures

Science knowledge

Development of the science knowledge scale consisted of infectious disease experts and middle school science teachers reviewing 25 pilot questions to ensure reflection of game content, scientific accuracy, and appropriateness for middle school. Questions were derived from the scientific process, content, and careers presented within the three career paths in Disease Defenders (Table 1, Appendix 1) and consisted of recall and application questions. All items were multiple-choice with three or four response options. Item-by-item analysis of pilot data from students not participating in the field test refined the scale down to 17 items. Within the 17 items there were two different question types: 13 related to scientific process knowledge and four related to general science knowledge. The internal consistency reliability (Cronbach’s α) was 0.63 for the science knowledge pretest and 0.79 for the posttest.

Attitudes

The attitude questions included three scales: attitudes toward science, satisfaction with the game, and game usability (see Appendix 1). All three scales have been previously used and validated (14, 15, 19). For each scale, participants read statements and rated their level of agreement on a five-point Likert scale from 1 = Strongly Disagree to 5 = Strongly Agree. The internal consistency reliability (Cronbach’s α) was 0.75 for both the pretest and posttest attitudes toward science scale, and 0.88 and 0.61 for satisfaction and usability scales, respectively (posttest only).

RESULTS

Of the original 1231 middle school students who consented, 375 were excluded from the analyses for the following reasons: identification numbers could not be matched from pretest to posttest, students left answers blank, or students reported not finishing the game. The final sample included 856 students: 489 in the experimental group and 367 in the control group.

Science knowledge acquisition

To assess science knowledge acquisition, mean test scores in both the experimental and control groups for the 17 science knowledge questions were calculated (Table 2). For the experimental group, the mean test score increased from 7.49 (44.1%) before playing Disease Defenders to 9.60 (56.5%) after playing the game. In contrast, the group mean test score in the control group slightly decreased from 6.68 before playing the control game to 6.35 after playing. However, the comparison of pretest scores (7.49 for experimental and 6.68 for the control) using an independent-samples t-test revealed that the scores were significantly different (t[854] = 4.16, p < 0.01).

TABLE 2.

Pretest, posttest, and adjusted posttest means for overall science knowledge questions for the experimental and control groups before and after playing the assigned MedMyst game.

Group (n)	Pretest Mean (SD)	Posttest Mean (SD)	Adjusted Posttest Mean (SD)
Experimental (489)	7.49 (3.10)	9.60 (3.95)	9.30 (2.92)
Control (367)	6.68 (2.40)	6.35 (2.40)	6.58 (2.08)

Given this difference, a General Linear Model (GLM) was used to compute posttest scores adjusted for these differences. Posttest scores were predicted by group, pretest scores, and the interaction of group and pretest scores, which takes into account the different slopes for the two groups. The pretest scores were centered (converted to deviation-scores) before the cross-product term was computed so that the effect of groups would represent the average effect of groups.

The resulting adjusted posttest scores are shown in Table 2; the experimental group mean score was adjusted from 9.60 to 9.30 and the control group mean score was adjusted from 6.35 to 6.58. The difference between the adjusted posttest means for the experimental and control group was statistically significant, F(1,852) = 223.70, p < 0.001 and the effect size was large d = 1.05 (6), indicating that students in the experimental group who played Disease Defenders learned significantly more than students in the control group who did not play the game.

Attitudes measures

To assess shifts in attitudes toward science, mean test scores in both the experimental and control groups for the four science attitudes questions were calculated. For the experimental group, the science attitudes mean score shifted only slightly in a positive direction from 3.98 (SD = 0.75) to 4.01 (SD = 0.76) (with 5.0 being most positive) after playing the game, and a paired-samples t-test revealed that the shift was not significant (t[488] = 1.21, p = 0.227). For the control group, the mean score shifted slightly in a negative direction from 3.99 (SD = 0.71) to 3.92 (SD = 0.77) after playing. A paired-samples t-test revealed that the negative shift was significant (t[366]=2.59, p < 0.05); however, the effect size was small d = 0.18.

Usability and satisfaction with the Disease Defenders intervention were measured posttest only in the experimental group with usability posttest group means at 3.75 (SD = 0.70) and satisfaction posttest group means at 3.93 (SD = 0.80), with 5.0 being the most positive for both scales.

The science knowledge acquisition and attitudes measures analyses capture the overall effects of the Disease Defenders intervention compared to a control group. Below we provide a more extensive analysis of the Disease Defenders intervention by examining differences by question type, grade level, and gender within the experimental group only.

Science knowledge acquisition by question type

To assess science knowledge acquisition by question type, mean scores for general science knowledge questions and scientific process knowledge questions were calculated separately. For general science knowledge questions, the mean score increased from 2.12 (SD = 1.13) to 2.32 (SD = 1.21) after playing the game, and a paired-samples t-test revealed the shift was significant (t[488] = 3.54, p < 0.01); however, the effect size was small d = 0.23. A second paired-samples t-test revealed a significant positive shift (t[488] = 18.70, p < 0.01) in mean scores for scientific process knowledge questions from 5.37 (SD = 2.50) to 7.43 (SD = 3.20) and a large effect size (d = 1.23), indicating that learning gains for the overall science knowledge scale were driven by the scientific process knowledge items.

Differences between genders

A repeated measures ANOVA was conducted in GLM to examine gender differences in science knowledge acquisition. Pretest and posttest were entered as the repeated measures and gender was entered as a between-subject variable. There was no interaction between gender and learning (F(1,487) = 3.51, p = 0.062), suggesting that Disease Defenders was equally effective for both boys and girls.

To examine changes in science attitudes as a function of gender, a similar analysis in GLM was conducted with pre- and posttest science attitudes entered as the repeated measures and gender as a between-subject variable. This analysis revealed no difference in attitudes toward science between genders (F(1,487) = 0.338, p = 0.561). Furthermore, an independent-samples t-test with female and male students revealed that neither satisfaction nor game usability ratings differed between genders (satisfaction: t(443) = −0.14, p = 0.07; game usability: t(477) = −0.102, p = 0.77).

Differences across grades

Grade-level differences in the efficacy, likability, and usability of Disease Defenders were also examined in the experimental group. Although differences in pretest science knowledge were expected, with higher grades having more baseline knowledge than lower grades, we expected that each grade would show an increase in science knowledge. To examine the effects of knowledge, a repeated measures ANOVA was conducted in GLM with pretest and posttest entered as the repeated measures and grade as a between-subject variable. The amount of learning was significantly different across grades (F(2,486) = 8.08, p < 0.01) (Fig. 3). Although paired-samples t-tests performed separately for each grade revealed that the increase in science knowledge scores between pretest and posttest is significant in all grades, the comparison of group means showed that knowledge gains were higher among sixth grade students (increased 16.9% from 6.02 to 8.90, t(49) = −7.7, p < 0.01; n = 50) and seventh grade students (increased 14.9% from 7.76 to 10.30, t(299) = 15.46, p < 0.01; n = 300) compared to eighth grade students (increased 8.5% from 7.42 to 8.87, t(138) = −5.57, p < 0.01; n = 139).

FIGURE 3.

Interaction between science knowledge scores and grade. Error bars represent standard errors. Grade six pretest: mean 6.02 (SD 2.07), posttest: mean 8.90 (SD 2.78). Grade seven pretest: mean 7.76 (SD 3.04), posttest: mean 10.30 (SD 3.93). Grade eight pretest: mean 7.42 (SD 3.38), posttest: mean 8.87 (SD 4.30).

A similar analysis in GLM was performed to examine science attitudes with pre- and posttest attitude assessments as the repeated measures and grade as a between-subject variable. The results showed a significant difference in attitudes toward science between grades (F(2,486) = 6.58, p < 0.01) (Fig. 4). Separate paired-samples t-tests revealed that the difference in attitudes is driven by a significant positive change of science attitude between preand posttest among sixth grade students (t(49) = −3.03, p < 0.01), whereas both seventh and eighth grade students’ science attitudes did not change significantly (grade seven: t(299) = 0.086, p = 0.921; grade eight: t(138) = −0.111, p = 0.912).

FIGURE 4.

Interaction between attitudes toward science and grade. 1 = Most Negative and 5 = Most Positive attitude toward science. Error bars represent standard errors. Grade six pretest: mean 3.93 (SD 0.96), posttest: mean 4.23 (SD 0.76). Grade seven pretest: mean 4.04 (SD 0.71), posttest: mean 4.04 (SD 0.75). Grade eight pretest: mean 3.87 (SD 0.75), posttest: mean 3.87 (SD 0.75).

Grade-level differences in satisfaction were examined with an ANOVA. Results showed a significant difference (F(2,486) = 6.22, p < 0.01) in satisfaction such that students in lower grades rated the game more positively than those in higher grades (sixth grade mean: 4.09 (SD 0.76), seventh grade mean: 3.99 (SD 0.77), eighth grade mean 3.73 (SD 0.85) on a five-point Likert scale with 5 being the most positive). There was no difference in usability ratings across grades (F(2,486) = 0.508, p = 0.602).

In summary, students in grades six and seven gained more knowledge after playing the game than eighth grade students, yet learning gains were significant in all grades. Moreover, although the shift in attitudes toward science in the experimental group as a whole was not significant, the analysis across grades revealed a difference, with a significant positive change in attitudes only among sixth graders, who also rated their satisfaction with the game more favorably than students in higher grades.

DISCUSSION

The results of this evaluation support the hypothesis that Disease Defenders is effective at producing scientific process knowledge gains among middle school students. The demonstrated learning gains, resulting from students interacting with the game for only one class period, have a large effect size and are in line with results from previous evaluations of similar computer-based games (2, 14, 15, 19–22, 27). Although the average posttest score in the experimental group was only 56.5%, perhaps combining the game with other instructional interventions would produce even higher knowledge gains. For this study, teachers were specifically asked to refrain from reinforcing or discussing any of the game’s content until after the posttest. Normally, one would expect teachers to use games such as Disease Defenders as part of a learning cycle. Often teachers use the games as part of the Engage or Explore phase in the 5E Model (4); however, the experimental design was structured to test the efficacy of the intervention without the benefit of classroom reinforcement. Future evaluations including control groups exposed to standard classroom instruction on the scientific process compared to experimental groups incorporating Disease Defenders plus standard classroom instruction would further elucidate the contribution of the game in teaching the scientific process.

Digital games could provide a way to help shape attitudes toward science. This variable is important because during middle school and high school, science attitudes tend to shift negatively among students (11, 12). Studies also show that students who in eighth grade indicate interest in science and science careers are more likely to enroll in science courses and earn STEM degrees (18, 31), so middle school is a good age to target for shifting science attitudes and fostering interest in science. This is particularly important in light of the concern that the United Sates is not preparing enough students in the areas of STEM (16). In contrast to positive learning gains, the analysis of attitudes toward science among the experimental groups as a whole did not reveal a positive shift. These results could be due to multiple reasons, the first being the relatively high initial predisposition toward science reported by students, which makes it difficult to shift attitudes even higher. Students reported mean pretest science attitudes scores at 3.98, which is higher than the 3.0 Likert scale mid-point (5.0 being the most positive), and, after playing the game, reported mean posttest scores only slightly higher at 4.01. Secondly, although the science attitudes scale has a high reliability and was validated in prior studies (14, 15, 19), it consisted of only four questions and may not have been sensitive enough to capture small changes in attitude. Finally, while other studies demonstrated positive attitude shifts after exposure to computer-based science instruction (14, 28), those interventions consisted of longer exposures. Perhaps the length of Disease Defenders (approximately 30 minutes) was too short to influence students’ attitudes toward science.

A surprising finding was a small negative shift in students’ science attitudes in the control group. A possible explanation could be that students in the control played a different science game from the MedMyst series that does not focus on the scientific process, yet the pretest and posttest questions focused on scientific process. It is possible that students may have become frustrated answering questions that had nothing to do with the content presented in the game they played. To prevent frustration, future study designs could expose the control group to the same intervention as the experimental group after the posttest instead of between pre- and posttest.

Even though statistics now show that males and females both play digital games equally (17), the gender gaps that have been reported in STEM areas (25) mandate the analysis of possible gender differences in the efficacy of digital games. No statistically significant differences were found in mean scores for attitudes, satisfaction, usability, and scientific process knowledge, indicating that Disease Defenders does not have a gender bias and is equally effective for males and females in middle school. These results are in line with the finding that both genders benefit from science-based digital games and multi-user virtual environments (1, 8, 13, 14, 20, 21).

Since scientific practices are reinforced in sixth, seventh, and eighth grades (23), Disease Defenders was designed to be equally beneficial to students across all middle school grades. However, the analysis of grade differences revealed that students in sixth grade and seventh grade learned more from the game than students in eighth grade. In addition to the difference in science knowledge acquisition, the intervention resulted in a significant positive shift of science attitudes among sixth grade students, while such a shift was not found in the higher grades. Together with the fact that sixth graders also rated their satisfaction with the game higher than older students, these findings suggest that Disease Defenders may be more efficacious and appealing in sixth and seventh grade. It is possible that eighth graders’ low satisfaction with the game may have influenced their attitudes. However, due to the widely varying sample sizes (sixth grade n = 50, seventh grade n = 300, and eighth grade n = 139) these results should be taken with caution. Further research is necessary to determine what makes games appealing to different age groups.

While the evaluation of Disease Defenders demonstrated grade-level differences, students at all levels in middle school demonstrated science knowledge gains. Unsolicited feedback indicates the broad appeal of MedMyst across many grades, including high school and college. With the number of textbooks under-representing the scientific process, Disease Defenders could provide an interactive tool for teachers to use as a means to reinforce the scientific process and foster scientific thinking.