Abstract
Background: FantaTraining ® is an app that simulates a football league. Each participant represents a team, and the game is played with the opposing team by answering a questionnaire. In the intervals between games, participants can practice by consulting the educational material (films, short texts, or slides) in the app. Various prizes are offered to the winners of the championship. In this study, we aimed to evaluate whether the use of the FantaTraining ® app could improve the learning of anesthesia trainees registered in an online obstetric anesthesia course.
Methods: The study involved 282 trainees in anesthesia, from five Italian universities, registered in the Online Obstetric Anesthesia Course (OOAC) who were given the app. They were randomly allocated into two groups according to whether the app had been enabled to allow the participant to play the league (study group, n=137), or not (control group, n=145). All the trainees underwent entry and final tests, consisting of the same 40 multiple choice questions, respectively before and after completing the OOAC course.
Results: There were no differences in the scores obtained in the pre-course test between the groups. The mean score obtained in the final test was significantly greater than that obtained in the entry one in both groups (P<0.05) but the final test score obtained by the participants of the study group was significantly greater than that obtained by the control group. (P<0.001), regardless of the university of origin and year of specialization. Trainees stated that the app had helped their study, improving understanding and motivation, without increasing the intensity of study.
Conclusions: Using the FantaTraining ® app greatly improved trainees’ final exam performance after the online obstetric anesthesia course. The FantaTraining ® app seems a promising tool to improve learning outcomes by strengthening learning behaviors and attitudes towards learning.
Keywords: gamification, app, anesthesia, education
Introduction
In the recent years, games have been reported to be able to increase learning outcomes in many fields of medical education, thus leading to the introduction of the term “gamification” ( Koivisto & Hamari, 2019; Seaborn & Fels, 2015). This term refers to the inclusion of game elements, such as points and rewards, to tasks as incentives for participants in order to facilitate learning ( Caponetto et al., 2014; Koivisto & Hamari, 2019; Majuri et al., 2018; Osatuyi et al., 2018).
There are many games and gamification options. Quizzes, for example, being one of the most common teaching gamification, allowing participants to verify their knowledge by using different platforms, such as on the web or apps. Different strategies have been employed: the challenge based ( Koivisto & Hamari, 2019), the immersion based, which involves the user in a story ( Concannon et al., 2019) and the social-based gamification, which is based on competition and collaboration ( Romero, 2017).
Gamified activities aim to improve the students’ internal and external motivation based on the role of rewards and incentives ( Richter et al., 2015). In addition, gamification has also been associated with the goal-setting theory. This theory explains that there are four factors which can affect the students’ performance: their commitment towards the goal, the feedback they receive, the complexity of the activity, and the situational limits ( Landers, 2014; Locke & Latham, 2002; Locke & Latham, 2006). Basically, gamification requires a challenge, some progress feedback, the knowledge of the levels of achievement, and some type of competition ( Huang & Hew, 2018). Another theory linked to gamification is the flow theory, where the good psychological and physical state of the participant may optimize enjoyment and engagement. Consistent with this theory, gamification would require specific and easily comprehensible goals, instantaneous feedback, accomplishment indicators, and a satisfactory balance between challenges, the participant’s skills, and the seeming value of the activity ( Huang & Hew, 2018).
Game accessible via mobile phones from the app store has been used to deliver simulation courses to medical students ( Smith & Holland, 2021).
Brain Refresher Lab ( https://www.brainrefreshlab.com) has recently developed an app called FantaTraining ®, (available on Apple store and Google Play) which is based on the gamification theories and that has been successfully used in marketing and business training. This app simulates a football league in which participants sign up and play matches with other registered participants to win the league. There is training between matches and prizes for the winners. Each participant represents a team, and the game is played with the opposing team by answering a questionnaire that checks the number of correct answers on a chosen topic. In the intervals between games, participants can practice by consulting the educational material (films, short texts, or slides) in the app. Various prizes are offered to the winners of the championship.
In this study we aimed to evaluate whether the use of the FantaTraining ® app could improve the learning of anesthesia trainees registered in an online obstetric anesthesia course.
Methods
Ethical statement
This study was carried out according to the Declaration of Helsinki. In our region, simulation centers do not have access to a formal ethical approval process, and it was not possible to submit the study under a different ethics committee in Italy. However, the study followed the Healthcare Simulationist Code of Ethics supported by the Society for Simulation in Healthcare ( Healthcare Simulationist Code of Ethics, 2018) The study was low-risk and non-clinical, with no direct contact with participants as all data collection was doing online. The trainees participating were volunteers, researchers ensured that those taking part in the research would not be caused distress, all the participants' personal and other data were completely anonymized, and all the investigators had no conflict of interest and were not involved in any of the participants' university teaching programs. Written informed consent was obtained from all participants prior to participation in the study.
Study design
The study took place online from October to December 2021.
It involved 300 trainees in anesthesia, from five Italian universities, registered on the Online Obstetric Anesthesia Course (OOAC) of the European School of Obstetric Anesthesia and Simulation Center (EESOA, Rome) who were offered voluntary enrollment in this study. This three-month, annual course consists of seven online lessons of 90 minutes each on subjects of obstetric anesthesia.
After registering in the EESOA OOAC and before the start of the lessons, all the anesthesia trainees who had given their consent to participate in the study were sent the FantaTraining ® app. For the purpose of the study the FantaTraining ® app was assembled in such a way that it contained, for each of the seven online lessons, both the additional learning materials (short video clips, tests, and slides) and the multiple choice quizzes representing the football matches to be played.
The app requires the participant to play a game (answer quizzes on the topics covered in the lesson in competition with another participant) in a set time interval between lessons for a predefined duration and leaves the participant free to consult the teaching material ("coaching") until shortly before the game. The app automatically calculates a provisional ranking for each game session, which is made known to the players, who thus know the score of the opponents they are challenging.
Among the trainees enrolled in the study 18 declined, and therefore 282 were randomized (simple randomization) and allocated into two groups according to whether the app had been enabled to allow the participant to play the league (study group, n=137), or not (control group, n=145).
In practice, the participants belonging to the study group were able to use the app both to consult further teaching materials and to play the championship, while those in the control group could only consult the teaching materials.
The trainees in the control group were told that the top five in the league would be offered a free simulation course. To encourage participation and play ( Richter et al., 2015), all those who completed the course were offered a book on obstetric anesthesia.
All the trainees underwent to entry and final tests, consisting of the same 40 multiple choice questions, respectively before and after completing the OOAC course. Every correct answer was given a score of one, therefore the maximum score obtainable was 40.
In addition, the study group participants completed a satisfaction questionnaire ( Table 1) after having finished the course and given the final test.
Table 1. Satisfaction questionnaire.
| Likert Scale (1-5):
1 Strongly disagree/ 2 Disagree / 3 Neutral/ 4 Agree/ 5 Strongly agree • Would you have studied with the same intensity without the App? (intensity) • Has the App been helpful in motivating you to study? (motivation) • Has the App been helpful to you as a study support? (study support) • Was the App helpful in understanding the study material? (understanding) |
Statistics
Statistical software R version 4.0.2 was used for data processing and the generation of tables and figures. For quantitative variables (test score and time) appropriate descriptive statistics (i.e., mean, standard deviation, median, quartiles) were calculated and results reported as mean ± SD or median ±MAD. Absolute and percentage frequency distributions were presented for qualitative variables (age, gender, university, and year of specialization). Kolmogorov-Smirnov and Shapiro-Wilk normality tests were applied to test the normality hypothesis.
Parametric statistical models such as repeated measure mixed effect analysis of variance (ANOVA) (post-hoc Tukey), and linear regression, were applied to dependent variables, and assumptions about residuals were tested for homoskedasticity, independence, and identical distribution.
For questionnaires that included Likert scale response modes, agreement indicators were calculated to assess the degree of agreement of the participants. There was maximum agreement towards the worst judgment when the indicator is equal to 0, while there was maximum agreement towards the best judgment when it is equal to 1 (disagreement among respondents returns an indicator equal to 0.5).
Power analysis was carried out on the primary endpoint of the study (mixed effect repeated measure ANOVA) which provided for a sample size of 220 to ensure a level of significance of 95% and a test power of 80%.
In addition, for the analysis of the questionnaire, setting a confidence level of 90% and a margin of error of 5%, a sample size of 268 residents was required.
Results
The mean age of the participants was 29 (±3) years (N=282). There were no differences in the scores obtained in the pre-course test between the groups ( Figure 1). The mean score obtained in the final test (after the end of the course) was significantly greater than that obtained in the first one (performed before starting the course) in both groups (P<0.05) but the final test score obtained by the participants of the study group was significantly greater than that obtained by the control group. (P<0.001) ( Figure 1), regardless of the university of origin and the year of specialization ( Velardo, 2022).
Figure 1. Mean scores before and after the course with or without using of the FantaTraining ® app.
Boxplot and distribution of the mean score before and after the course with (study group) or without (control group) playing the championship. The horizontal line displays 25 th, 50 th, and 75 th percentile, the red point and vertical bar indicate the mean and confidence interval.
There were no differences between the groups in the time elapsed to perform the test ( Figure 1).
Concerning the analysis of the participants who had played the games, the median number of games played was eight out of a possible total of 13.
By applying a linear regression model, we observed that the mean score of the games played, and the percentage of matches won were positively correlated with the time of use of the app: the model estimated that for every hour of application use, the percentage of games won can increase by 15% (P<0.05) and the average score can increase by 3.6 (P<0.05) ( Figure 2).
Figure 2. Linear regression model of the correlation between the average score, matches won, and the time of use of the FantaTraining ® app in the study group.
Results of the questionnaire are reported in the Table 2. Trainees stated that the app helped their study, improving understanding and motivation, without increasing the intensity of study.
Table 2. Frequency distribution and agreement indicator of questionnaire answers (N=282).
| ITEM | Strongly
disagree |
Disagree | Neutral | Agree | Strongly
agree |
Agreement
indicator |
|---|---|---|---|---|---|---|
| Understanding | 1% | 2% | 13% | 43% | 41% | 0,87 |
| Intensity | 14% | 32% | 35% | 14% | 4% | 0,46 |
| Motivation | 1% | 7% | 17% | 37% | 39% | 0,83 |
| Support to study | 1% | 2% | 13% | 34% | 50% | 0,88 |
Discussion
The current young doctors’ generation utilizes games and simulations to create collaborative learning environments. The so called mobile-learning has the potential for social interactivity in gaming scenarios and increased practice problem-solving on different engaging and visually stimulating platforms. Rather than the traditional education and information delivery methods, users can have individualized experiences, track their progress, and study alone or with colleagues adhering to their specific time and location limitations ( Becker, 2019).
In this study the primary source of knowledge was a ‘traditional’ online course, and we employed the app as a teaching support to be used between the online lessons.
Using the FantaTraining ® app greatly improved the trainees’ final exam performance after the online obstetric anesthesia course. Our results are in line with the theories that gamification improves trainees’ engagement and motivation by adding game elements such as instant feedback, earning rewards and tracking challenge completion. Gamification also adjusts the way the brain processes new knowledge through releasing bite-sized portions of information, audio-visual stimulation, short time-lapses and repetitive patterns ( Sailer & Homner, 2020; Shurui Bai et al., 2020; van Gaalen et al., 2021).
The positive results obtained with the FantaTraining app may be explained by the theory of gamified learning ( Landers, 2014) which suggests that instructional content impacts learning outcomes as well as learners’ behavior . According to this theory, the goal of gamification is to alter a contextual learner’s behavior or attitude (e.g., engagement), to improve pre-existing instruction.
In addition to an increased learning performance, we also noted high levels of satisfaction reported by our users: trainees affirmed that the app significantly helped their study improving both understanding and motivation, without increasing the intensity of study and this may certainly be considered an additional advantage of the use of this app. This participant opinion was also confirmed by the results obtained by our regression model, which was able to predict a significant improvement in learning performance by increasing the amount of time of use of the app.
Limitations
Our study has some limitations. It is a preliminary study and, although the sample size is more than sufficient to justify the results, the use of the FantaTraining ® app should be extended to a larger sample in order to better evaluate its effects over time. We hope that it will be included in the educational programs of the universities that have joined the study, so as to be able to better highlight any criticalities of the app, which will probably be highlighted only with its routine use.
Conclusion
In conclusion, the FantaTraining ® app seems a promising tool to improve learning outcomes by strengthening learning behaviors and attitudes towards learning, and we hope the results of our study may contribute to a better comprehension of the role of new learning methods in health profession education. Further studies are also needed with students from other medical specialties to confirm our positive findings. In particular, it would be interesting to investigate the exclusive use of FantaTraining ® as the primary substitute resource for online classes.
Data availability
Underlying data
Harvard Dataverse: Improving learning by playing with an app: The FantaTraining® experience with anesthesia. https://doi.org/10.7910/DVN/TXDS2E. ( Velardo, 2022).
This project contains the following underlying data:
DB Fantatraining 1. (Anonymized results for control and study group participants on Fanta-Training database.)
ANOVA by university. (ANOVA results considering university)
ANOVA by years of specialization. (ANOVA results considering year of specialization)
Data are available under the terms of the Creative Commons Zero “No rights reserved” data waiver (CC0 1.0 Public domain dedication).
Acknowledgements
Thanks are due to all the trainees who accepted to be involved in the study and to their tutors. In particular, thanks are due to Antonello Giarratano and Andrea Cortegiani (University of Palermo), Elena Bignami (University of Parma), Luca Carenzo (Humanitas University of Milan), Marco Scorzoni and Gian Luigi Gonella (UCSC University of Rome), Rosanna Vaschetto (University of Novara) and Alessandra Lauretta (Bellinzona and Lugano Hospitals).
Funding Statement
The author(s) declared that no grants were involved in supporting this work.
[version 1; peer review: 2 approved]
References
- Becker D: Mobile Gaming for Medical Students and Learners. J Electr Res Med Lib. 2019;16(3–4):1–6. 10.1080/15424065.2019.1692319 [DOI] [Google Scholar]
- Caponetto I, Earp J, Ott M: Gamification and education: A literature review. 8th European Conference on Games Based Learning (Academic Conferences International Limited),2014;50–57. Reference Source [Google Scholar]
- Concannon BJ, Esmail S, Roberts MR: Head-Mounted Display Virtual Reality in Post-secondary Education and Skill Training. Front Educ. 2019;4:80. 10.3389/feduc.2019.00080 [DOI] [Google Scholar]
- Huang B, Hew KF: Implementing a Theory-Driven Gamification Model in Higher Education Flipped Courses: Effects on out-of-Class Activity Completion and Quality of Artifacts. Comput Educ. 2018;125:254–272. 10.1016/j.compedu.2018.06.018 [DOI] [Google Scholar]
- Koivisto J, Hamari J: The rise of motivational information systems: a review of gamification research. Int J Inf Manage. 2019;45:191–210. 10.1016/j.ijinfomgt.2018.10.013 [DOI] [Google Scholar]
- Landers RN: Developing a Theory of Gamified Learning: Linking Serious Games and Gamification of Learning. Simul Gaming. 2014;45:752–768. 10.1177/1046878114563660 [DOI] [Google Scholar]
- Locke EA, Latham GP: Building a Practically Useful Theory of Goal Setting and Task Motivation. A 35-Year Odyssey. Am Psychol. 2002;57(9):705–717. 10.1037/0003-066X.57.9.705 [DOI] [PubMed] [Google Scholar]
- Locke EA, Latham GP: New Directions in Goal-Setting Theory. Curr Dir Psychol Sci. 2006;15(5):265–268. 10.1111/j.1467-8721.2006.00449.x [DOI] [Google Scholar]
- Majuri J, Koivisto J, Hamari J: Gamification of education and learning: a review of empirical literature. In: Proceedings of the 2nd International GamiFIN Conference, GamiFIN, 2018, CEUR-WS, (Finland),2018. Reference Source [Google Scholar]
- Osatuyi B, Osatuyi T, De La Rosa R: Systematic review of gamification research in IS education: A multi-method approach. Communications of the Association for Information Systems. 2018;42(1):95-124. 10.17705/1CAIS.04205 [DOI] [Google Scholar]
- Park CS, Murphy TF, the Code of Ethics Working Group: Healthcare Simulationist Code of Ethics.2018. Reference Source [Google Scholar]
- Richter G, Raban DR, Rafaeli S: Studying gamification: the effect of rewards and incentives on motivation. In: Gamification in Education and Business.201: eds Reiners T., Wood L. (Cham: Springer),2015;21–46. 10.1007/978-3-319-10208-5_2 [DOI] [Google Scholar]
- Romero M: Competitive, collaborative, and cooperative play. In: The SAGE Encyclopedia of Out-of-School Learning.ed. Peppler K. (New York, NY: SAGE Publications, Inc.),2017;115–116. 10.4135/9781483385198.n51 [DOI] [Google Scholar]
- Sailer M, Homner L: The Gamification of Learning: a Meta-analysis. Educ Psychol Rev. 2020;32:77–112. 10.1007/s10648-019-09498-w [DOI] [Google Scholar]
- Seaborn K, Fels DI: Gamification in theory and action: a survey. Int J Hum Comput Stud. 2015;74:14–31. 10.1016/j.ijhcs.2014.09.006 [DOI] [Google Scholar]
- Shurui Bai, Hew KF, Huang B: Does gamification improve student learning outcome? Evidence from a meta-analysis and synthesis of qualitative data in educational contexts. Educ Res Review. 2020;30:100322. 10.1016/j.edurev.2020.100322 [DOI] [Google Scholar]
- Smith G, Holland J: Game On! Meeting the Mission of Gamification to Teach Human Factor Skills. Int J Health Sim. 2021;1(Suppl 1):A60–A61. Reference Source [Google Scholar]
- van Gaalen AEJ, Brouwer J, Schönrock-Adema J, et al. : Gamification of health professions education: a systematic review. Adv Health Sci Educ Theory Pract. 2021;26(2):683–711. 10.1007/s10459-020-10000-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Velardo M: Improving learning by playing with an app: The FantaTraining® experience with anesthesia. Harvard Dataverse, V1,2022. 10.7910/DVN/TXDS2E [DOI] [PMC free article] [PubMed] [Google Scholar]