Abstract
Background
Smartphone-based learning, or mobile learning (m-learning), has become a popular learning-and-teaching strategy in educational environments. Blended learning combines strategies such as m-learning with conventional learning to offer continuous training, anytime and anywhere, via innovative learning activities.
Objective
The main aim of this work was to examine the short-term (ie, 2-week) effects of a blended learning method using traditional materials plus a mobile app—the iPOT mobile learning app—on knowledge, motivation, mood state, and satisfaction among undergraduate students enrolled in a health science first-degree program.
Methods
The study was designed as a two-armed, prospective, single-blind, randomized controlled trial. Subjects who met the inclusion criteria were randomly assigned to either the intervention group (ie, blended learning involving traditional lectures plus m-learning via the use of the iPOT app) or the control group (ie, traditional on-site learning). For both groups, the educational program involved 13 lessons on basic health science. The iPOT app is a hybrid, multiplatform (ie, iOS and Android) smartphone app with an interactive teacher-student interface. Outcomes were measured via multiple-choice questions (ie, knowledge), the Instructional Materials Motivation Survey (ie, motivation), the Profile of Mood States scale (ie, mood state), and Likert-type questionnaires (ie, satisfaction and linguistic competence).
Results
A total of 99 students were enrolled, with 49 (49%) in the intervention group and 50 (51%) in the control group. No difference was seen between the two groups in terms of theoretical knowledge gain (P=.92). However, the intervention group subjects returned significantly higher scores than the control group subjects for all postintervention assessed items via the motivation questionnaire (all P<.001). Analysis of covariance (ANCOVA) revealed a significant difference in the confusion and bewilderment component in favor of the intervention group (P=.01), but only a trend toward significance in anger and hostility as well as total score. The intervention group subjects were more satisfied than the members of the control group with respect to five out of the six items evaluated: general satisfaction (P<.001), clarity of the instructions (P<.01), clarity with the use of the learning method (P<.001), enough time to complete the proposed exercises (P<.01), and improvement in the capacity to learn content (P<.001). Finally, the intervention group subjects who were frequent users of the app showed stronger motivation, as well as increased perception of greater gains in their English-language competence, than did infrequent users.
Conclusions
The blended learning method led to significant improvements in motivation, mood state, and satisfaction compared to traditional teaching, and elicited statements of subjective improvement in terms of competence in English.
Trial Registration
ClinicalTrials.gov NCT03335397; https://clinicaltrials.gov/ct2/show/NCT03335397
Keywords: learning; teaching; students, health occupations; mobile apps; education
Introduction
Smartphone-based learning or mobile learning (m-learning), a term that highlights the type of device used, has become a popular learning-and-teaching strategy. It has been defined as the ability to access educational resources, tools, and materials anytime and from anywhere, using a mobile device, such as a smartphone [1]. With respect to health professionals and their education, Dunleavy et al describe m-learning as “any intervention using handheld, mobile devices connected through wireless connections to deliver educational content to pre and postregistration health professionals in order to extend the reach of learning and teaching beyond physical space and distance” [2]. M-learning thus provides a self-directed learning environment that affords continuous access to knowledge, information, and practice tools.
M-learning is being increasingly used by undergraduate and postgraduate students undertaking specialty or continuous training in the health sciences [2,3]. A recent review and meta-analysis showed that m-learning is equally or more effective than traditional learning in this field, although the authors also comment on the need for further research into its value [2]. Given the rapid evolution of mobile technologies in general, and of m-learning in particular, up-to-date evidence is required for the effectiveness of these strategies to be determined.
Motivation is an important factor in learning and performance [4]. Motivation levels can vary depending on learning-and-teaching style. A positive relationship has been reported between motivation and academic performance and learning, while a negative association is associated with dropping out from education [5]. Thus, motivation can be increased if the learning-and-teaching style is appropriate [6,7]. M-learning incorporates different activity options, which students can choose depending on their personal learning preferences or circumstances. This can encourage motivation, engagement, and learning success, positively influencing mood status [6,7].
New technologies may, however, also have a negative effect on learning. A recent study [8] highlights how students believed that new technologies in the classroom might affect their concentration and ability to learn. However, other investigations have found that students using electronic devices during a lecture returned results in learning that were similar to those who did not use these devices [9]. In other work, using electronic devices as learning tools while listening to a teacher was reported to not be distracting [10]. Clearly, the impact of the use of m-learning needs to be further investigated.
Blended learning that combines m-learning with conventional learning to seek the benefits of both [11,12] can also improve the educational experience [13], allowing conventionally taught material to be revisited whenever and wherever the student wishes [12,14]. Previous investigations into blended learning have returned positive results [15,16]. The main aim of this work was to examine the short-term effects of a blended learning method using traditional materials plus a mobile app on specific outcomes among students enrolled in a health science first-degree program. It was hypothesized that, compared to traditional learning alone, a 2-week blended learning program would improve knowledge uptake, student motivation, mood, and satisfaction and would improve competence in English.
Methods
Study Design and Sample Size
This study was a two-arm, prospective, single-blind, randomized controlled trial (ClinicalTrials.gov identifier: NCT03335397). The study subjects were 99 students in their first year of a health sciences degree at the Faculty of Health Sciences, University of Granada, Spain.
The sample size calculation was based on a previous study [17] and a demand of 90% power to detect a difference of 1.70 points in knowledge gain (see Main Outcomes section), while assuming a type 1 error (alpha) of 5% and a type 2 error (beta) of 10%, determined using G*Power software, version 3.1.9.3 (Heinrich-Heine-Universität Düsseldorf), for Mac OS X [18]. The required sample size was estimated at 40 subjects each for the intervention and control groups. Contemplating a dropout rate of 25%, as it is common for health science students to migrate to medicine in the first semester of their first academic year, 50 subjects were enrolled per group.
Recruitment
Subjects were recruited via a talk given on the first day of the course unit entitled Basic Health Science, a unit lasting one semester. It was emphasized that no assessment made during the trial would have any effect on final grades and that participation was totally voluntary. To be included, the subjects had to (1) possess basic skills in handling mobile apps, (2) have a smartphone running either Android operating system (OS) or iOS software, (3) install the iPOT mobile learning app, (4) be enrolled in the above course unit, and (5) have a basic knowledge of English, accredited or not. In addition, all subjects had to provide informed consent to be included. Students repeating the unit were excluded.
Randomization
Subjects who met the inclusion criteria were randomly assigned to either the intervention group (ie, blended learning, involving traditional learning plus m-learning via the use of the iPOT app) or the control group (ie, traditional on-site learning). Figure 1 shows a flowchart of the recruitment and randomization process. Subjects were allocated to these groups via the computer generation of a random number list and the subsequent production of a subject allocation sequence using SPSS statistics software, version 22.0 (IBM Corp). The subjects then met with the evaluator who would conduct the forthcoming assessments; this evaluator was blinded to all group assignments.
Figure 1.
Flow diagram of the recruitment and randomization process. OS: operation system.
Experimental Procedure
This study was conducted over 2 weeks in the first semester of the 2017-2018 academic year. For both groups, the educational program involved 13 lessons on basic health science. The subjects of both groups were progressively introduced to conventional learning materials (ie, books, PowerPoint presentations, and journals available in the university library). The subjects in the intervention group, however, also received the iPOT app to reinforce the educational program; it provided no additional material. Both the intervention and control subjects were free to reinforce the taught information using any of the traditional sources available. All subjects in the intervention group received a QR (Quick Response) code to provide them access to the Apple App Store [19] or the Google Play store [20] as required. Subjects then installed the iPOT app. A lecturer involved in the project was present at this time to help solve any technical problems that might arise. All the intervention group subjects were provided a personal password linked to their institutional email account that allowed them to access the app, but were also assured each person's log-in information remained confidential and nonexchangeable with subjects in the control group; the system itself was enabled to detect irregular connections. However, given that students in the control group might borrow the devices of the intervention group members in order to try the app out, all subjects were requested to respect the protocol and told they would have unlimited access to the app at the end of the study. The iPOT app included a permanently available video tutorial (see Multimedia Appendix 1), accessible through the Help tab, to assist users in familiarizing themselves with the app and its operation. In addition, a WhatsApp group was established for the 2 weeks of the experimental period in order to resolve any incidents that might arise. Two smartphones were available to loan to students who wanted to participate but who had devices with compatibility problems.
M-learning lessons were enabled in the app over the last 2 weeks of the teaching period, allowing the intervention group subjects to review the information previously taught. Students were instructed to review each lesson via the six learning modalities available—quizzes (multiple-choice questions), alphabet soup (word search), hangman, hieroglyphics, puzzles (word sorting), and word-coupling activities—and were encouraged to use them; differences in use time were accounted for (see The iPOT Mobile Learning App: Internal Design and Technical Specifications section). It is important to highlight that during these 2 weeks, the control group subjects were encouraged to review the taught material via the above-mentioned books, PowerPoint presentations, and journals.
The iPOT Mobile Learning App: Internal Design and Technical Specifications
The iPOT app was designed by a research group composed of lecturers in health science and an internship student of computer engineering, all at the University of Granada. The app is a hybrid, multiplatform (iOS and Android), smartphone app with an interactive teacher-student interface (see Figure 2). Its architecture involves Firebase, a Google cloud-based development platform that handles user authentication, file storage, and database management. The app interface gives users access to different game modes and allows them to complete different activities and find platform-related information. The app also has an administration console, developed using Angular and Angular Material development tools (Google), that gives administrators the possibility of managing users, all app modules, game-related data, and graphic resources by adding an abstraction layer between the administrator and Firebase. For the deployment of the administration console, it was decided to use Firebase Hosting, a static, rapid hosting service that includes a Secure Sockets Layer (SSL) by default, ensuring safe access.
Figure 2.
Top-level view of the iPOT app system.
The user interface shows the user profile, allowing access to personal data and providing the opportunity to change one's nickname; ranking (ie, position in terms of game scores; designed to help increase motivation via a sense of competition); and a start button to allow users to begin playing. As an incentive, each time a correct answer was given in any of the app's modalities, points were added to the user's score. Each user's scores, as well as the number of modules completed, were recorded via the system.
Outcome Measures
Knowledge and mood state were assessed for both the intervention and control groups before and after the 2-week experimental period. Motivation, satisfaction, and English-language competence—the latter only measured in the intervention group—were assessed after this period.
Knowledge
Theoretical knowledge was evaluated at baseline (ie, the beginning of the 2-week study period) and at the end of the study period using 20 multiple-choice questions [21]. This tool has been used in other educational studies [22,23]. Scores were recorded on an 11-point scale, ranging from 0 to 10, with higher values representing better outcomes. This test was developed by two external lecturers who were not involved in the study.
Motivation
Motivation was assessed using the Instructional Materials Motivation Survey (IMMS) [24]. The terminology used by the tool was adapted to the learning methods employed in this study. The survey can be used to estimate the motivation and attitudes of students in situations of self-directed learning (ie, printed, virtual, or online courses). It contains 36 questions in four domains: attention, relevance, confidence, and satisfaction. Each question is scored on a 5-point Likert scale; therefore, total scores range from 36 to 180. Higher scores denote higher levels of learning motivation. The tool has been shown to be highly reliable (intraclass correlation coefficient [ICC]>.96) [25] and has been previously used in Spanish populations [26,27].
Mood State
Mood is a short-term state of feeling; it may fluctuate within minutes or over days. Unlike emotions, moods are more transient, often unrelated to external events, and are of varying intensity [28]. Subject mood state was assessed via the Profile of Mood States (POMS) questionnaire [29], validated for use in Spain [30]. This self-reported survey contains 65 questions, each scored on an increasing 5-point Likert scale, ranging from 0 to 4, that together measure six mood components: (1) tension and anxiety (nine items that refer to an increase in musculoskeletal tension, somatic tension, and observable psychomotor manifestations); (2) depression and melancholy (15 items that reflect a depressed state along with a feeling of sadness or guilt); (3) anger and hostility (12 items that represent feelings of anger and antipathy toward others); (4) vigor and activity (eight items that allude to a state of euphoria or high energy); (5) fatigue and inertia (seven items that refer to a mood of depression or a low level of exercise); and (6) confusion and bewilderment (seven items referring to disorientation or multiplicity of thought). The total average score of all the components (ie, the Total Mood Disturbance [TMD] score) was recorded. There are no cutoff points in any domain. Subjects were asked to fill in the questionnaire indicating how they felt lately. Lower values indicate a better general mood state [31]. The test is reliable; the ICC is close to .90 for all six components [29].
Satisfaction
Subjects' satisfaction with the learning methods to which they were exposed was evaluated via a questionnaire based on that reported by Brewer et al [32], with six items that assess satisfaction variables: general satisfaction, the clarity of instructions, whether the final assessment reflected the course syllabus, clarity with the use of the learning method, whether there was enough time to complete the proposed exercises, and improvement in the capacity to learn content. Each item was scored on an increasing 5-point Likert scale, ranging from 1 to 5. Higher values represent better outcomes. Subjects were also asked to report the number of hours per day, as well as the number of days per week, that they used the app and which mode of the app they preferred.
Linguistic Competence
The subjective improvement in English-language competence (ie, a general feeling of improvement and of improvements in written comprehension, written expression, vocabulary, and global perception) was measured at the end of the study via an ad hoc questionnaire, scoring answers on an increasing 5-point Likert scale, ranging from 1 to 5. Higher values represent better outcomes. To determine the subjects' prior English-language competence, they were asked about any official English certificates they possessed before starting the study.
Statistical Analysis
The normal distribution of data was verified using the Kolmogorov-Smirnov test. Results are presented as mean (SD) values. The Student t test and Mann-Whitney U test or the chi-square test and Fisher exact test were used as required to examine differences in continuous and categorical variables, respectively, between the two groups (eg, sociodemographic characteristics, knowledge, and mood state) and in postintervention differences (eg, motivation, satisfaction, and English-language competence).
Differences between the groups in terms of the preintervention-to-postintervention change in knowledge and mood state were analyzed by repeated-measures analysis of covariance (ANCOVA), adjusting for the effects of those variables showing significant differences between groups at baseline (ie, the covariate POMS-vigor and activity). Missing values were few (<5% of the total number); in such cases, these can be considered missing at random and inconsequential [33]. For this reason, a list-wise deletion method was chosen; no multiple imputation was necessary. Significance was set at P<.05. All calculations were performed using SPSS statistics software, version 22.0 (IBM Corp), and Stata statistical software, release 14 (StataCorp).
Ethical Approval
The study was performed in accordance with the ethical standards of the appropriate national and institutional research committees, and adhering to the Declaration of Helsinki [34]. Ethical approval was granted as required by Spanish Law 223/2004, of February 6, establishing subject confidentiality under the terms of Spanish Law 3/2018, of December 5 [35]. The study was also approved by the Quality, Innovation and Planning Unit of the University of Granada, Spain (Plan FIDO UGR 2016-2018). For ethical reasons, once the experimental phase of the project was complete, the app was made available to all subjects who took part.
Results
User Statistics
A total of 99 students were enrolled, with 49 (49%) in the intervention group and 50 (51%) in the control group. The mean age of the students was 19.76 years (SD 2.74) in the intervention group versus 20.00 years (SD 3.98) in the control group (P=.97). A total of 38 of the 49 subjects (78%) in the intervention group were women compared to 35 out of 50 subjects (70%) in the control group (P=.39). A total of 37 of the 49 subjects (76%) in the intervention group used mobile phones with the Android OS compared to 31 out of 50 subjects (62%) in the control group (P=.14). No significant differences were seen between the members of the two groups in terms of their baseline levels of English (P=.69) or baseline knowledge of the material to be taught (P=.43) (see Table 1). English levels were determined according to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment [36]. The intervention group lost one subject (1/49, 2%) to follow-up—a dropout due to the subject's phone running an old OS that did not support the iPOT app—leaving a final initial sample size of 48. Another 2 subjects were lost from both groups due to incompletion of the final assessments (see Figure 1).
Table 1.
Sociodemographic characteristics of the study subjects at baseline: the beginning of the 2-week study period.
| Characteristic | Control group (n=50) | Intervention group (n=49) | ||
| Age (years), mean (SD) | 20.00 (3.98) | 19.76 (2.74) | ||
| Gender, n (%) |
|
|
||
|
|
Male | 15 (30) | 11 (22) | |
|
|
Female | 35 (70) | 38 (78) | |
| English level, n (%)a |
|
|
||
|
|
Not certified | 25 (50) | 22 (45) | |
|
|
A2 (elementary) | 4 (8) | 2 (4) | |
|
|
B1 (low intermediate) | 14 (28) | 15 (31) | |
|
|
B2 (high intermediate) | 5 (10) | 9 (18) | |
|
|
C1 (advanced) | 2 (4) | 1 (2) | |
| Mobile system, n (%) |
|
|
||
|
|
iOS | 19 (38) | 12 (24) | |
|
|
Android operating system (OS) | 31 (62) | 37 (76) | |
| Baseline knowledge of subjects (scoreb), mean (SD) | 4.38 (1.21) | 4.57 (1.22) | ||
aEnglish levels were determined according to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment [36]. A2: Basic User, Waystage; B1: Independent User, Threshold; B2: Independent User, Vantage; C1: Proficient User, Effective Operational Proficiency.
bScores ranged from 0 (no knowledge) to 10 (highest level of knowledge).
Main Outcomes
At the end of the experimental period, no significant difference was seen between the two groups in terms of theoretical knowledge gain (intervention group mean score 0.51 [SD 1.19] vs control group mean score 0.49 [SD 1.10]; F1=0.008, P=.92) (see Table 2). A significant difference was also not seen in the end-of-study total scores (mean score 5.16 [SD 1.30] vs mean score 4.90 [SD 1.24], respectively; t92=-1.019, P=.31). The covariate POMS-vigor and activity did not influence the results.
Table 2.
Effect of the different learning methods on knowledge gain.
| Answers at study time points | Control group (n=48), mean (SD) | Intervention group (n=46), mean (SD) | P valuea | |
| Correct answers |
|
|
.92 | |
|
|
Baselineb | 8.81 (2.43) | 9.30 (2.37) |
|
|
|
Postintervention | 9.79 (2.48) | 10.33 (2.61) |
|
|
|
Difference | 0.98 (2.20) | 1.02 (2.37) |
|
| Wrong answers |
|
|
.92 | |
|
|
Baseline | 11.19 (2.43) | 10.70 (2.38) |
|
|
|
Postintervention | 10.21 (2.48) | 9.67 (2.61) |
|
|
|
Difference | –0.98 (2.20) | –1.02 (2.37) |
|
| Total score |
|
|
.92 | |
|
|
Baseline | 4.41 (1.21) | 4.65 (1.19) |
|
|
|
Postintervention | 4.90 (1.24) | 5.16 (1.30) |
|
|
|
Difference | 0.49 (1.10) | 0.51 (1.19) |
|
aRepeated-measures analysis of covariance (ANCOVA) was used to examine the differences between groups.
bBaseline refers to the beginning of the 2-week study period.
Table 3 shows that the intervention group subjects returned significantly higher scores than the control group subjects for all items assessed postintervention by the IMMS (IMMS_attention: t92=–12.223; IMMS_relevance: t92=–8.315; IMMS_confidence: t92=–7.731; and IMMS_satisfaction: t92=–10.631; P<.001 for all). Their IMMS total motivation scores were also significantly different (mean 136.02 [SD 19.25] vs mean 84.94 [SD 25.37], respectively; t92=–10.963, P<.001).
Table 3.
Learning motivation in the intervention and control groups.
| Learning motivation variablea,b | Control group (n=48), mean (SD) | Intervention group (n=46), mean (SD) | P valuec |
| Attention (12-60) | 26.06 (9.64) | 47.59 (7.20) | <.001 |
| Relevance (9-45) | 24.06 (6.58) | 34.17 (5.08) | <.001 |
| Confidence (9-45) | 22.04 (6.96) | 31.83 (5.13) | <.001 |
| Satisfaction (6-30) | 12.77 (4.55) | 22.44 (4.25) | <.001 |
| Total IMMSa score (36-180) | 84.94 (25.37) | 136.02 (19.25) | <.001 |
aLearning motivation was measured using the Instructional Materials Motivation Survey (IMMS).
bEach category is presented with the possible range of its score in parentheses.
cThe Student t test was used to examine the differences between groups. Significance was set at P<.05.
Table 4 reflects the mood state of the subjects in each group before and after the intervention. ANCOVA revealed a significant difference between the change in confusion and bewilderment for the control and intervention group subjects (mean score 3.67 [SD 8.66] vs mean score –0.50 [SD 6.62], respectively; F1=6.826, P=.01). A trend toward significance was also seen between the groups in terms of the change in anger and hostility (control group mean score 2.21 [SD 7.97] vs intervention group mean score –1.07 [SD 8.45]; F1=3.735, P=.05) and total score (control group mean score –1154.17 [SD 3652.10] vs intervention group mean score –17.39 [SD 2767.05]; F1=2.875, P=.09). No significant differences between groups were seen in terms of the change in tension and anxiety (F1=1.964, P=.16), depression and melancholy (F1=0.983, P=.32), vigor and activity (F1=0.181, P=.67), and fatigue and inertia (F1=0.233, P=.63). Adjustment for the covariate POMS-vigor and activity in ANCOVA revealed this variable to influence the change in confusion and bewilderment (F1=7.889, P=.006), anger and hostility (F1=4.493, P=.03), and total score (F1=4.603, P=.03). The covariate did not influence the rest of results.
Table 4.
Effect of the different learning methods on the Profile of Mood States (POMS) total and subscale scores.
| Mood state | Control group (n=48), mean (SD) | Intervention group (n=46), mean (SD) | P valuea | ||||
| Tension and anxiety |
|
|
.16 | ||||
|
|
Baselineb | 45.58 (8.43) | 46.78 (9.70) |
|
|||
|
|
Postintervention | 46.15 (9.45) | 45.00 (10.09) |
|
|||
|
|
Difference | 0.56 (8.88) | –1.78 (7.21) |
|
|||
| Depression and melancholy |
|
|
.32 | ||||
|
|
Baseline | 47.38 (5.55) | 47.39 (6.82) |
|
|||
|
|
Postintervention | 49.21 (7.64) | 47.93 (8.64) |
|
|||
|
|
Difference | 1.83 (6.55) | 0.54 (6.04) |
|
|||
| Anger and hostility |
|
|
.05 | ||||
|
|
Baseline | 51.77 (9.52) | 52.67 (8.95) |
|
|||
|
|
Postintervention | 53.98 (11.79) | 51.61 (10.24) |
|
|||
|
|
Difference | 2.21 (7.97) | –1.07 (8.45) |
|
|||
| Vigor and activity |
|
|
.67 | ||||
|
|
Baseline | 54.25 (7.30) | 57.43 (6.05) |
|
|||
|
|
Postintervention | 51.96 (7.55) | 54.67 (6.77) |
|
|||
|
|
Difference | –2.29 (5.80) | –2.76 (4.82) |
|
|||
| Fatigue and inertia |
|
|
.63 | ||||
|
|
Baseline | 47.75 (7.02) | 46.59 (6.79) |
|
|||
|
|
Postintervention | 48.73 (9.49) | 46.80 (7.45) |
|
|||
|
|
Difference | 0.98 (9.24) | 0.22 (5.51) |
|
|||
| Confusion and bewilderment |
|
|
.01 | ||||
|
|
Baseline | 38.54 (7.38 ) | 39.65 (8.02) |
|
|||
|
|
Postintervention | 42.21 (8.43) | 39.15 (7.64) |
|
|||
|
|
Difference | 3.67 (8.66) | –0.50 (6.62) |
|
|||
| Total score |
|
|
.09 | ||||
|
|
Baseline | –17677.08 (2899.91) | –17565.22 (3453.13) |
|
|||
|
|
Postintervention | –18831.25 (4052.59) | –17582.61 (4088.16) |
|
|||
|
|
Difference | –1154.17 (3652.10) | –17.39 (2767.05) |
|
|||
aRepeated-measures analysis of covariance (ANCOVA) was used to examine the differences between groups. Significance was set at P<.05; a trend toward significance was defined as .05≤P<.10.
bBaseline refers to the beginning of the 2-week study period.
Figure 3 shows the results of the global satisfaction survey with the learning method used. The intervention group subjects were more satisfied than the members of the control group in terms of all items evaluated: general satisfaction with the learning method (Strongly agreed and Agreed in the intervention group were 30% and 65%, respectively, vs 2% and 10% in the control group; P<.001), clarity of the instructions (Strongly agreed and Agreed in the intervention group were 57% and 35%, respectively, vs 27% and 31% in the control group; P<.01), clarity with the use of the learning method (Strongly agreed and Agreed in the intervention group were 30% and 54%, respectively, vs 2% and 29% in the control group; P<.001), enough time to complete the proposed exercises (Strongly agreed and Agreed in the intervention group were 9% and 17%, respectively, vs 8% and 4% in the control group; P<.01), and improvement in the capacity to learn content (Strongly agreed and Agreed in the intervention group were 11% and 57%, respectively, vs 0% and 17% in the control group; P<.001). No difference was seen between the groups with respect to satisfaction regarding how the final assessment reflected the course syllabus (Strongly agreed and Agreed in the intervention group were 11% and 43%, respectively, vs 6% and 31% for the control group; P=.06).
Figure 3.
Satisfaction among the intervention and control arms.
The intervention group subjects used the app an average of 3.85 (SD 1.51) days per week. The quiz option was the most popular, used by 83% (38/46) of the subjects, followed by word coupling (29/46, 63%) and alphabet soup (20/46, 43%). In further analysis, the median number of days spent using the app per week (median 4 days) was used to categorize students as infrequent (<4 days/week) and frequent users (≥4 days/week). Motivation among the frequent users was significantly higher than among the infrequent users (IMMS_attention: frequent, mean score 50.04 [SD 5.17] vs infrequent, mean score 44.40 [SD 8.29], t92=–2.669, P=.01; IMMS_confidence: frequent, mean score 33.77 [SD 4.87] vs infrequent, mean score 29.30 [SD 4.38], t92=–3.222, P<.01; IMMS_satisfaction: frequent, mean score 24.00 [SD 3.03] vs infrequent, mean score 20.40 [SD 4.78], t92=–3.114, P<.01; and IMMS_total: frequent, mean score 142.85 [SD 15.52] vs infrequent, mean score 127.15 [SD 20.34], t92=–2.971, P<.01). No significant difference was detected, however, between the users in the subgroups in terms of IMMS_relevance (frequent, mean score 35.04 [SD 4.24] vs infrequent, mean score 33.05 [SD 5.92], t92=–1.328, P=.19). The frequent users also perceived their general linguistic competence in English to have significantly improved through the use of the app (P=.03), although this was not reflected when examining specific domains: written expression (P=.57), written comprehension (P=.18), and vocabulary (P=.11).
Discussion
Principal Findings
The blended learning method led to significant improvements in motivation, mood state, and satisfaction compared to traditional teaching. Moreover, frequent users of the app showed stronger motivation and perceived greater gains in their English-language competence than did infrequent users. However, the results suggest the intervention strategy provided no benefit in terms of knowledge absorbed. The latter finding is consistent with the results of two previous randomized controlled studies evaluating online learning methods [22,23]. In contrast, Chuang et al, whose work had a similar design to this investigation (ie, blended learning vs regular lecturing) and was of the same duration (ie, 2 weeks), reported significant differences in favor of blended learning [37]. It has been suggested that the use of audiovisual material, as employed in the latter study, might generate better learning outcomes [38].
Comparison With Prior Work
In this work, motivation was measured by the validated IMMS tool. Higher scores were recorded for all motivation dimensions in the intervention group. Keller defines motivation as an innate characteristic of students, but also indicates that it can be influenced by external factors such as the instructional method used [39]. The innovative design for reviewing classes described in this work may be responsible for the increased motivation among the intervention group subjects. The use of augmented reality among middle school students [26], and of 3D computer environments among undergraduate students [40], have returned similar improvements in motivation. A recent study [41] reports that a mobile augmented reality (mAR) app boosted motivation in 35 students at four vocational education and training institutes over a period of 20 days. Real-time feedback, the degree of success achieved (ie, ranking), time on task (ie, frequent vs infrequent users), and learning outcomes (ie, theoretical tests) were some of the main predictors of improved learning in this latter study. The greatest difference between the subjects in the groups of this study was seen with respect to IMMS_attention (ie, the interest and curiosity of students in the learning process), which was significantly greater among the intervention subjects. The app's attractive interface, enjoyable quizzes, and user-friendliness may be partially responsible for this result.
The intervention group subjects appeared to be more involved in the learning process; they returned a significant change in two of the six mood state components assessed, which was reflected in the total score. Moreover, anger and hostility as well as confusion and bewilderment improved in the intervention group subjects, while these became worse (ie, increased scores) among those in the control group. The use of the app might thus have helped improve negative mood states. Pekrun et al [42] stated that academic achievement is a complex interaction between emotions, engagement, and performance, and this influences student self-appraisal. Flexible and creative learning strategies facilitate positive academic emotions, while more rigid strategies may spur negative emotions [42]. The “flow” concept proposes that an optimal educational experience may exist when students feel that a task is meaningful and, while perhaps challenging, that they are up to the challenge [43]. Other authors [17,44] who tested mAR interventions (ie, 30 minutes and 45 minutes in length), using a variation of the POMS questionnaire to measure change in mood state, only showed intrasubject changes.
Finally, satisfaction was greater among the intervention group subjects. It seems clear that the role of m-learning was perceived as an appropriate complement to traditional lecturing. With the development of advanced technologies, smartphones can be an effective learning tool for students. The possibility of reviewing classes whenever and wherever one likes, with tailored feedback (ie, scores for the games and quizzes), may have promoted the transfer of knowledge into their short-term and even long-term memories [45].
Limitations
This study suffers from the limitation that it lasted only 2 weeks. This short time frame was due to external factors such as a funder-imposed deadline. Further, the iPOT app was created for use as a post-teaching period review tool, not as a study tool in itself. No difference was detected between the amount of knowledge absorbed by the members of the two groups. However, if, as the results show, the use of the app motivates students, a significant difference in what is learned might be expected after a longer period of use. Further work should investigate this.
Conclusions
Students who received instruction via blended learning involving m-learning showed greater motivation, a better mood state, and greater satisfaction than those who received traditional lectures alone, although no difference was seen in terms of the amount of knowledge absorbed. Longer studies are needed to determine whether the improvements in these factors persist and whether they eventually translate into more knowledge being absorbed.
Acknowledgments
We are grateful to Adrian Burton for assistance with the English language. The authors are also grateful to the students who participated in the study. This work was funded by a grant from the Educational Innovation Unit of the University of Granada, Spain (PID 16-54). Additional funding was provided by the University of Granada, Plan Propio de Investigación 2016, Excellence actions: Units of Excellence; Unit of Excellence on Exercise and Health (UCEES).
Abbreviations
- ANCOVA
analysis of covariance
- ICC
intraclass correlation coefficient
- IMMS
Instructional Materials Motivation Survey
- mAR
mobile augmented reality
- m-learning
mobile learning
- OS
operating system
- POMS
Profile of Mood States
- QR
Quick Response
- SSL
Secure Sockets Layer
- TMD
Total Mood Disturbance
- UCEES
Unit of Excellence on Exercise and Health
Appendix
iPOT video tutorial.
CONSORT-EHEALTH checklist (V 1.6.1).
Footnotes
Conflicts of Interest: None declared.
References
- 1.GSMA . mLearning: A Platform for Educational Opportunities at the Base of the Pyramid. London, UK: GSMA Development Fund; 2010. Nov, [2020-04-29]. https://www.gsma.com/mobilefordevelopment/wp-content/uploads/2012/04/mlearningaplatformforeducationalopportunitiesatthebaseofthepyramid.pdf. [Google Scholar]
- 2.Dunleavy G, Nikolaou CK, Nifakos S, Atun R, Law GC, Tudor Car L. Mobile digital education for health professions: Systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res. 2019 Feb 12;21(2):e12937. doi: 10.2196/12937. https://www.jmir.org/2019/2/e12937/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Guo P, Watts K, Wharrad H. An integrative review of the impact of mobile technologies used by healthcare professionals to support education and practice. Nurs Open. 2016 Apr;3(2):66–78. doi: 10.1002/nop2.37. http://europepmc.org/abstract/MED/27708817. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Kusurkar RA, Ten Cate TJ, van Asperen M, Croiset G. Motivation as an independent and a dependent variable in medical education: A review of the literature. Med Teach. 2011 Apr 25;33(5):e242–e262. doi: 10.3109/0142159x.2011.558539. [DOI] [PubMed] [Google Scholar]
- 5.Wouters A, Croiset G, Galindo-Garre F, Kusurkar RA. Motivation of medical students: Selection by motivation or motivation by selection. BMC Med Educ. 2016 Jan 29;16:37. doi: 10.1186/s12909-016-0560-1. https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-016-0560-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Kilis S. Impacts of mobile learning in motivation, engagement and achievement of learners: Review of literature. Gaziantep Univ J Soc Sci. 2013;12(2):375–383. ISSN: 1303-0094 http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=023AD76D52E3D7CE8A9186E183C94A98?doi=10.1.1.1029.4362&rep=rep1&type=pdf. [Google Scholar]
- 7.Orsini C, Binnie VI, Wilson SL. Determinants and outcomes of motivation in health professions education: A systematic review based on self-determination theory. J Educ Eval Health Prof. 2016;13:19. doi: 10.3352/jeehp.2016.13.19. doi: 10.3352/jeehp.2016.13.19. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Attia N, Baig L, Marzouk YI, Khan A. The potential effect of technology and distractions on undergraduate students' concentration. Pak J Med Sci. 2017;33(4):860–865. doi: 10.12669/pjms.334.12560. http://europepmc.org/abstract/MED/29067054. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Nalliah R, Allareddy V. Students distracted by electronic devices perform at the same level as those who are focused on the lecture. PeerJ. 2014;2:e572. doi: 10.7717/peerj.572. doi: 10.7717/peerj.572. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Pew Research Center . Millennials: A Portrait of Generation Next. Confident. Connected. Open to Change. Washington, DC: Pew Research Center; 2010. Feb, [2020-04-29]. https://www.pewresearch.org/wp-content/uploads/sites/3/2010/10/millennials-confident-connected-open-to-change.pdf. [Google Scholar]
- 11.Delialioglu O, Yildirim Z. Design and development of a technology enhanced hybrid instruction based on MOLTA model: Its effectiveness in comparison to traditional instruction. Comput Educ. 2008 Aug;51(1):474–483. doi: 10.1016/j.compedu.2007.06.006. [DOI] [Google Scholar]
- 12.Feist M, Ciccarelli M, McFerron BA, Molleston JP. Methods and effects of a case-based pediatric gastroenterology online curriculum. J Pediatr Gastroenterol Nutr. 2013;56(2):161–165. doi: 10.1097/mpg.0b013e31825677d7. [DOI] [PubMed] [Google Scholar]
- 13.Graham CR. Blended learning systems: Definition, current trends, and future directions. In: Bonk CJ, Graham CR, editors. The Handbook Of Blended Learning: Global Perspectives, Local Designs. San Francisco, CA: Pfeiffer Publishing; 2005. pp. 3–21. [Google Scholar]
- 14.Yapici İÜ, Akbayin H. The effect of blended learning model on high school students' biology achievement and on their attitudes towards the internet. Turk Online J Educ Technol. 2012;11(2):228–237. https://files.eric.ed.gov/fulltext/EJ989031.pdf. [Google Scholar]
- 15.Liu Q, Peng W, Zhang F, Hu R, Li Y, Yan W. The effectiveness of blended learning in health professions: Systematic review and meta-analysis. J Med Internet Res. 2016 Jan 04;18(1):e2. doi: 10.2196/jmir.4807. https://www.jmir.org/2016/1/e2/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Alamro AS, Schofield S. Supporting traditional PBL with online discussion forums: A study from Qassim Medical School. Med Teach. 2012 Mar 12;34(sup1):S20–S24. doi: 10.3109/0142159x.2012.656751. [DOI] [PubMed] [Google Scholar]
- 17.Albrecht U, Folta-Schoofs K, Behrends M, von Jan U. Effects of mobile augmented reality learning compared to textbook learning on medical students: Randomized controlled pilot study. J Med Internet Res. 2013 Aug 20;15(8):e182. doi: 10.2196/jmir.2497. https://www.jmir.org/2013/8/e182/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Faul F, Erdfelder E, Lang A, Buchner A. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods. 2007 May;39(2):175–191. doi: 10.3758/BF03193146. [DOI] [PubMed] [Google Scholar]
- 19.Apple App Store. [2020-05-15]. iPOT https://apps.apple.com/es/app/ipot/id1291766873.
- 20.Google Play. [2020-05-08]. iPOT https://play.google.com/store/apps/details?id=com.ipot.ipotMobileLearning&hl=es.
- 21.Considine J, Botti M, Thomas S. Design, format, validity and reliability of multiple choice questions for use in nursing research and education. Collegian. 2005 Jan;12(1):19–24. doi: 10.1016/s1322-7696(08)60478-3. [DOI] [PubMed] [Google Scholar]
- 22.Fernández-Lao C, Cantarero-Villanueva I, Galiano-Castillo N, Caro-Morán E, Díaz-Rodríguez L, Arroyo-Morales M. The effectiveness of a mobile application for the development of palpation and ultrasound imaging skills to supplement the traditional learning of physiotherapy students. BMC Med Educ. 2016 Oct 19;16(1):274. doi: 10.1186/s12909-016-0775-1. https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-016-0775-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Arroyo-Morales M, Cantarero-Villanueva I, Fernández-Lao C, Guirao-Piñeyro M, Castro-Martín E, Díaz-Rodríguez L. A blended learning approach to palpation and ultrasound imaging skills through supplementation of traditional classroom teaching with an e-learning package. Man Ther. 2012 Oct;17(5):474–478. doi: 10.1016/j.math.2012.04.002. [DOI] [PubMed] [Google Scholar]
- 24.Keller JM. Strategies for stimulating the motivation to learn. Nonprofit Manag Leadersh. 1987 Oct;26(8):1–7. doi: 10.1002/pfi.4160260802. [DOI] [Google Scholar]
- 25.Keller JM. Motivational Design for Learning and Performance: The ARCS Model Approach. New York, NY: Springer Science+Business Media; 2010. [Google Scholar]
- 26.Di Serio Á, Ibáñez MB, Delgado Kloos C. Impact of an augmented reality system on students' motivation for a visual art course. Comput Educ. 2013 Oct;68:586–596. doi: 10.1016/j.compedu.2012.03.002. [DOI] [Google Scholar]
- 27.Castaño-Garrido C, Maiz-Olazabalaga I, Garay-Ruiz U. Design, motivation and performance in a cooperative MOOC course. Comunicar. 2015 Jan;22(44):19–26. doi: 10.3916/c44-2015-02. https://files.eric.ed.gov/fulltext/ED564097.pdf. [DOI] [Google Scholar]
- 28.Berger BG, Motl RW. Exercise and mood: A selective review and synthesis of research employing the profile of mood states. J Appl Sport Psychol. 2000 Mar;12(1):69–92. doi: 10.1080/10413200008404214. [DOI] [Google Scholar]
- 29.McNair DM, Lorr M, Droppleman LF. Manual for the Profile of Mood States. 3rd edition. San Diego, CA: Educational and Industrial Testing Service (EdITS); 1992. [Google Scholar]
- 30.Balaguer SI, Fuentes I, Meliá JL, García-Merita ML, Pérez Recio G. El perfil de los estados de ánimo (POMS): Baremo para estudiantes Valencianos y su aplicación en el contexto deportivo. Rev Psicol Deporte. 1993;4(2):39–52. https://www.researchgate.net/profile/Isabel_Balaguer/publication/288166199_El_perfil_de_los_estados_de_animo_POMS_Baremo_para_estudiantes_valencianos_y_su_aplicacion_en_el_contexto_deportivo/links/56dd7e3e08aedf2bf0c84bd2/El-perfil-de-los-estados-de-animo-POMS-Baremo-para-estudiantes-valencianos-y-su-aplicacion-en-el-contexto-deportivo.pdf. [Google Scholar]
- 31.Lane AM, Jackson A, Terry PC. Preferred modality influences on exercise-induced mood changes. J Sports Sci Med. 2005 Jun 01;4(2):195–200. http://europepmc.org/abstract/MED/24431976. [PMC free article] [PubMed] [Google Scholar]
- 32.Brewer ZE, Ogden WD, Fann JI, Burdon TA, Sheikh AY. Creation and global deployment of a mobile, application-based cognitive simulator for cardiac surgical procedures. Semin Thorac Cardiovasc Surg. 2016;28(1):1–9. doi: 10.1053/j.semtcvs.2016.02.006. [DOI] [PubMed] [Google Scholar]
- 33.Schafer JL. Multiple imputation: A primer. Stat Methods Med Res. 1999 Mar;8(1):3–15. doi: 10.1177/096228029900800102. [DOI] [PubMed] [Google Scholar]
- 34.World Medical Association World Medical Association Declaration of Helsinki: Ethical principles for medical research involving human subjects. JAMA. 2013 Nov 27;310(20):2191–2194. doi: 10.1001/jama.2013.281053. [DOI] [PubMed] [Google Scholar]
- 35.Spanish Government Agencia Estatal Boletín Oficial del Estado. 2018. Dec 06, [2020-04-29]. Ley Orgánica 3/2018, de 5 de diciembre, de Protección de Datos Personales y garantía de los derechos digitales https://www.boe.es/eli/es/lo/2018/12/05/3.
- 36.Council of Europe . Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge, UK: Cambridge University Press; 2001. [2020-05-08]. https://rm.coe.int/1680459f97. [Google Scholar]
- 37.Chuang Y, Lai F, Chang C, Wan H. Effects of a skill demonstration video delivered by smartphone on facilitating nursing students' skill competencies and self-confidence: A randomized controlled trial study. Nurse Educ Today. 2018 Jul;66:63–68. doi: 10.1016/j.nedt.2018.03.027. [DOI] [PubMed] [Google Scholar]
- 38.Mayer RE. Designing multimedia instruction in anatomy: An evidence-based approach. Clin Anat. 2020 Jan;33(1):2–11. doi: 10.1002/ca.23265. [DOI] [PubMed] [Google Scholar]
- 39.Keller JM. Development and use of the ARCS model of instructional design. J Instr Dev. 1987 Sep;10(3):2–10. doi: 10.1007/bf02905780. [DOI] [Google Scholar]
- 40.Hu A, Shewokis PA, Ting K, Fung K. Motivation in computer-assisted instruction. Laryngoscope. 2016 Aug;126 Suppl 6:S5–S13. doi: 10.1002/lary.26040. [DOI] [PubMed] [Google Scholar]
- 41.Bacca J, Baldiris S, Fabregat R, Kinshuk Insights into the factors influencing student motivation in augmented reality learning experiences in vocational education and training. Front Psychol. 2018;9:1486. doi: 10.3389/fpsyg.2018.01486. doi: 10.3389/fpsyg.2018.01486. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Pekrun R, Goetz T, Titz W, Perry RP. Academic emotions in students' self-regulated learning and achievement: A program of qualitative and quantitative research. Educ Psychol. 2002 Jun;37(2):91–105. doi: 10.1207/s15326985ep3702_4. [DOI] [Google Scholar]
- 43.Csikszentmihalyi M. Flow: The Psychology of Optimal Experience. New York, NY: HarperCollins Publishers; 2008. [Google Scholar]
- 44.Noll C, von Jan U, Raap U, Albrecht U. Mobile augmented reality as a feature for self-oriented, blended learning in medicine: Randomized controlled trial. JMIR Mhealth Uhealth. 2017 Sep 14;5(9):e139. doi: 10.2196/mhealth.7943. https://mhealth.jmir.org/2017/9/e139/ [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Stenlund T, Sundström A, Jonsson B. Effects of repeated testing on short- and long-term memory performance across different test formats. Educ Psychol. 2014 Sep 02;36(10):1710–1727. doi: 10.1080/01443410.2014.953037. [DOI] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
iPOT video tutorial.
CONSORT-EHEALTH checklist (V 1.6.1).