Determining the effectiveness of a cell phone-based student response system

Abstract

Objectives

This study aimed to explore the responses and understanding of students and teachers about the cell phone-based Socrative® application. Additionally, we compared the academic performance of the groups using Socrative with the one group that did not make use of this application.

Methods

During the 50 min of each endocrine physiology lecture, traditional teaching took 40 min, whereas the last 10 min were reserved for Socrative. Following the lecture, students completed small Socrative-based quizzes on their smartphones. At the end of the module, students and teachers were asked for feedback and the students’ test performances were evaluated. Data were examined by the SPSS version 20 for frequencies and gender/academic performance comparison.

Results

As many as 87% of the students responded positively to Socrative; 85% felt that the activity was fun, 84% were more actively engaged, and 71% felt more motivated. Furthermore, 90% of the students agreed that this exercise enhanced their learning. Most students agreed that Socrative helped them enhance peer-to-peer and class discussions. Students’ attitudes towards Socrative activity did not vary with respect to their gender or total screen time per day. Approximately 99% of the students preferred multiple-choice questions to true/false and short answers. Instructors perceived Socrative as an easy-to-use tool to generate discussions and assess the degree of understanding of their students. Socrative significantly enhanced exam performance.

Conclusion

Course instructors are encouraged to incorporate mobile-based applications in their lectures, which may make students’ learning more active, effective, and enjoyable, without increasing institutional expenses.

Introduction

The use of technology and active learning techniques in the medical school curriculum has led to improvements in students’ engagement and critical thinking skills.1, 2, 3 Didactic lecturing may make learners feel bored due to its monotonous style.^4,5 Active learning techniques transform a teacher-centered approach into a student-centered one and ensure higher-order thinking,³ engagement,⁶ and collaboration.⁷ Active Learning includes a variety of teaching and learning techniques, student response systems (SRS) being one of them.⁸

SRSs provide an idea about the degree of students' understanding of lecture contents. Based on this information, lectures can be modified, and misconceptions can be sorted out by student–student or student-teacher discussions.⁹ SRSs improve students’ attendance, attention, participation, engagement, and interaction; stimulate peer and class discussion; increase long-term retention of knowledge¹⁰; and enhance exam performance.¹¹ SRSs have been used in various fields of medical education, such as microbiology,¹² obstetrics and gynecology,^13,14 pediatrics,¹⁵ physiology,¹⁶ and oral and maxillofacial radiology.¹⁷

However, the use of SRSs is not trouble-free. One problem arising with SRSs is related to their initial costs; this prompted the search for a free alternative. Smartphones have become an integral part of everybody's lives in the modern era and have made technology more accessible and less expensive. Students have shown favourable responses to the use of mobile phones as a learning tool.^18,19

The mobile-based Socrative software can be a free alternative to SRSs. Socrative is a convenient way for the teachers to get instant, real-time, formative assessment data. Through this software, an instructor can construct quizzes consisting of a variety of question types, such as multiple-choice (MCQ), true/false, closed-ended, and open-ended, and share these quizzes with students or other instructors. By selecting the option ‘Instant Feedback’, instructors can see which questions have been answered incorrectly by most of the class and clear the misconception in an effective way.²⁰ Hence, Socrative allows to instantly judge a class' understanding of a concept.

A previous study revealed that the incorporation of Socrative improved student interaction and collaborative learning among community college students at King Abdulaziz University in KSA.²¹ Similarly, another study showed that Socrative enhanced active engagement and interactive learning in the classroom in pharmaceutical sciences in Qatar.²² Yet, to the authors’ knowledge, there is no study available in the literature that has assessed the use of Socrative among medical students in KSA. In view of this, the aim of the present research was to answer the following questions:

• 1.How do students feel about their experience of using Socrative in their classes?
• 2.Are there any gender differences in attitudes toward using Socrative?
• 3.Is there any association between students' attitude toward Socrative with their total daily screen time?
• 4.How do instructors feel about their experience of incorporating Socrative through smartphones in their classes?
• 5.Is there any difference in students' performance in examination questions with and without Socrative?

Materials and Methods

The study was conducted at the Physiology Department of the College of Medicine in our university, between November 2018 and June 2019. All procedures complied with the ethical standards of the institutional Review Board of our University and with the 1964 Declaration of Helsinki and its later amendments. The study participants were third-year medical students (18–20 years old; 80 females and 70 males). The context of the study was an endocrinology module that consisted of 10 lectures of 50 min each. The classrooms were equipped with free wi-fi connections, and students were asked to download the Socrative Student app to their smartphones before starting the module. The instructors downloaded the Socrative Teacher app to their smartphones and completed the registration. After their successful registration, instructors were provided with a class code number.

For each lecture, instructors prepared short online quizzes consisting of multiple choice, true/false, and short answer questions in the Socrative app, covering salient aspects of the lecture. During the 50 min of each physiology class, traditional teaching took 40 min, whereas the last 10 min of each lecture were reserved for the Socrative activity. Students logged into the Socrative Student app, joined their instructor's class, and completed the quiz posted by the instructor for that day's lecture. Students had been briefed at the beginning of the course that the scores obtained from those quizzes would have no consequences in the grade book. Students were encouraged to participate in order to make lectures more interactive and generate discussions.

Gathering students' views about the Socrative SRS was the primary objective of this study. During the last class of the endocrinology module, a survey administered through Google Forms was used to obtain students' feedback on incorporating Socrative into the lectures. Items were measured using a Likert scale from one (strongly disagree) to five (strongly agree). Students' questionnaires were based on those used for other studies.^21,23 The reliability and internal consistency of the questionnaire were confirmed through a test-retest technique and Cronbach's alpha, respectively, and found to be high (Cronbach's alpha 0.79). Items included in the survey evaluated students' liking, fun/enjoyment, engagement, participation, motivation, interactions, understanding, and learning experience with Socrative. Students were also asked about the time spent on smartphones, tablets, laptops, etc. for academic and non-academic (playing video games, internet surfing, watching movies/serials, and using social media) purposes per day. We interviewed the instructors involved in teaching the module with Socrative and asked about the advantages, disadvantages, limitations, and recommendations on the use of Socrative software in the classroom through ‘open comments’. Their comments were included in the results where a consensus was found.

Participation in the Socrative activity at the end of the lecture was mandatory, whereas completing the Google Forms survey was purely voluntary.

The end-of-term exam consisted of multiple-choice questions (MCQs), extended matching questions, and short answer questions. For this research, students served as a self-control group and their performance in MCQs of the Endocrinology Module (module with Socrative) were compared with their performance in the remaining MCQs (modules without Socrative). None of the MCQs from Socrative class quizzes were repeated in the end-of-term exam. The item analysis conducted at the end of the examination revealed that each of the MCQs fulfilled the criteria of being able to discriminate between higher and lower scorers (point biserial > 0.2), and the distractors in each MCQ were plausible to exclude the effect of confounding factors, such as guessing. The reliability of the exam was 0.93.

Data were entered on SPSS Version 20. We used an independent sample t-test to compare means between genders and employed a Pearson correlation to find the association between students' attitudes and total screen time daily. Percentages of the students among the entire class, and high scorers and low scorers who answered correctly between ‘Module with Socrative’ and ‘Modules without Socrative’ were compared using independent sample t-tests. Significance was set at p < 0.05 (students were categorised into high and low scorers based on overall exam scores.)

Results

Of a total course enrolment of 150, 128 surveys were completed, yielding a survey response rate of 85%. For ease of presentation, five-point Likert scale responses were grouped together according to the following: strongly agree and agree as ‘Agree’; strongly disagree and disagree as ‘Disagree’ in Table 1. 86.7% of students enjoyed using Socrative for answering questions from the lecture, compared to 3.1% student who did not. 88.3% of students felt that the use of Socrative online quizzes made the learning experience more enjoyable and fun compared to traditional lectures, livened up the lecture, and helped them stay interested throughout class time. In addition, 84% of students felt more actively engaged, 73% participated more, and 71% felt more motivated. Around 90% of the students agreed that this experience helped them focus on key knowledge in the class, understand the subject matter in depth, and enhanced their learning and overall learning experience. The majority of students agreed that Socrative gave them an opportunity to have more discussions with their classmates (87.5% vs 5.4%), with their lecturer (77.3% vs 5.5%), made them feel valuable through the interactions (72.6% vs 6.2%) and improved their personal relationships with their classmates and lecturer (61.7% vs 10.9%). More than 94% of the students were satisfied with Socrative and recommended its use in other courses as well, compared to a small proportion of students (only 0.8%) who did not agree.

Table 1.

Students' Experiences in Using Socrative.

Statements	Agree N (%)	Neutral N (%)	Disagree N (%)
1. I liked using Socrative to answer questions during lectures.	111 (86.7)	13 (10.2)	4 (3.1)
2. It made the learning experience more enjoyable/fun compared to traditional lectures.	113 (88.3)	13 (10.2)	2 (0.16)
3. Using Socrative helps liven up the lecture.	109 (85.2)	18 (14.1)	1 (0.8)
4. It allowed me to stay interested during class time.	111 (86.7)	14 (10.9)	3 (2.4)
5. I felt more actively engaged in class lectures.	108 (84.4)	17 (13.3)	3 (2.4)
6. It encouraged me participate more in class.	94 (73.4)	28 (21.9)	6 (4.7)
7. It motivates me to be more prepared for class.	91 (71.1)	28 (21.9)	9 (7.1)
8. It allowed me to focus on key knowledge in the class.	117 (90.4)	8 (6.3)	3 (2.3)
9. It helped me understand the subject matter more compared to non-technology classes (deep understanding.)	111 (86.8)	14 (10.9)	3 (2.4)
10. It enhanced my learning in this course.	115 (89.8)	9 (7)	4 (3.1)
11. It improved the overall learning experience.	112 (87.5)	14 (10.9)	2 (1.6)
12. Using Socrative gave me the opportunity to have more discussions with classmates.	112 (87.5)	9 (7)	7 (5.4)
13. Using Socrative gave me the opportunity for more discussions with the lecturer.	99 (77.3)	22 (17.2)	7 (5.5)
14. This way, my classmates and faculty interactions made me feel valuable.	94 (72.6)	27 (21.1)	8 (6.2)
15. I feel that this method improved my personal relationships with my classmates and lecturer.	79 (61.7)	35 (27.3)	14 (10.9)
16. I would recommend the use of Socrative in other classes within the medical college.	118 (92.2)	9 (7)	1 (0.8)
17. Overall, I am satisfied with the use of Socrative in lectures.	121 (94.6)	6 (4.7)	1 (0.8)

N = number of the study participants.

Male and female students did not differ significantly in their opinion about the Socrative experience in any of the items, as shown in Table 2. Total screen time/day was calculated by summing up the time spent on smartphones, tablets, laptops, etc. for academic and non-academic purposes. Pearson correlations did not reveal any significant associations between total daily screen time and any of the items in the student survey (Table 3). Table 4 shows instructors’ feedback about the Socrative experience.

Table 2.

Gender-based Comparison of Socrative Responses.

	Statements	Average Response Males (65) (Mean ± SD)	Average Response Females (63) (Mean ± SD)	p-value (t test)
1.	I liked using Socrative to answer questions during lectures.	4.40 ± 0.92	4.68 ± 0.59	0.38
2.	It made the learning experience more enjoyable/fun compared to traditional lectures.	4.37 ± 0.84	4.43 ± 0.69	0.66
3.	Using Socrative helps liven up the lecture.	4.32 ± 0.85	4.48 ± 0.72	0.27
4	It allowed me to stay interested during class time.	4.25 ± 0.99	4.43 ± 0.78	0.25
5.	I felt more actively engaged in class lectures.	4.32 ± 0.87	4.38 ± 0.85	0.70
6.	It encouraged me participate more in class.	4.06 ± 1.10	4.19 ± 0.98	0.49
7.	It motivates me to be more prepared for class.	3.98 ± 1.15	4.10 ± 0.99	0.56
8.	It allowed me to focus on key knowledge in the class.	4.46 ± 0.75	4.54 ± 0.69	0.54
9.	It helped me understand the subject matter more compared to non-technology classes (deep understanding).	4.45 ± 0.77	4.35 ± 0.85	0.50
10.	It enhanced my learning in this course.	4.40 ± 0.84	4.46 ± 0.74	0.67
11.	It improved the overall learning experience.	4.45 ± 0.87	4.41 ± 0.75	0.82
12.	Using Socrative gave me the opportunity to have more discussions with classmates.	4.29 ± 1.07	4.57 ± 0.73	0.09
13.	Using Socrative gave me the opportunity of more discussions with the lecturer.	4.14 ± 0.03	4.25 ± 0.90	0.50
14.	In this way, my classmates and faculty interactions made me feel valuable.	4.14 ± 0.06	4.11 ± 0.90	0.88
15.	I feel that this method improved my personal relationships with my classmates and lecturer.	3.88 ± 0.14	3.83 ± 0.13	0.80
16.	I would recommend the use of Socrative in other classes within the medical college.	4.51 ± 0.73	4.57 ± 0.67	0.61
17.	Overall, I am satisfied with the use of Socrative in lectures.	4.51 ± 0.71	4.56 ± 0.62	0.69

Table 3.

Relationship Between Students’ Perception About Socrative and Daily Screen Time.

		Daily screen time
		r	p-value
1.	I liked using Socrative to answer questions during lectures.	0.151	0.09
2.	It made the learning experience more enjoyable/fun compared to traditional lectures.	−0.02	0.98
3.	Using Socrative helps liven up the lecture.	0.096	0.29
4	It allowed me to stay interested during class time.	0.025	0.78
5.	I felt more actively engaged in the class lectures.	0.151	0.09
6.	It encouraged me participate more in class.	0.06	0.49
7.	It motivates me to be more prepared for class.	0.147	0.10
8.	It allowed me to focus on key knowledge in the class.	0.029	0.75
9.	It helped me understand the subject matter more compared to non-technology classes (deep understanding)	0.136	0.13
10.	It enhanced my learning in this course.	0.092	0.31
11.	It improved the overall learning experience.	0.070	0.44
12.	Using Socrative gave me the opportunity to have more discussions with classmates	−0.149	0.098
13.	Using Socrative gave me the opportunity of more discussions with the lecturer.	−0.050	0.58
14.	This way, my classmates and faculty interactions made me feel valuable.	−0.049	0.58
15.	I feel that this method improved my personal relationships with my classmates and lecturer	−0.073	0.42
17.	I would recommend the use of Socrative in other classes within the medical college.	0.126	0.16
18.	Overall, I am satisfied with the use of Socrative in lectures.	0.098	0.28

Table 4.

Instructors’ Feedback About Socrative.

1.	Advantages	➢Instant feedback ➢Provokes discussions ➢Empowers the teacher to assess students' understanding ➢Discover misconceptions and their remedies ➢Enhances students' engagement
2.	Disadvantages	➢Increase in session preparation time ➢Shortens the time used for the delivery of the lecture contents. ➢Time consuming
3.	Limitations	Strong wi-fi signals
4.	Recommendations	Future use for attendance, assignments, homework, pre-post lectures quizzes to assess levels of understanding

Percentages of the students among the entire class, and upper and lower third scorers who answered correctly at the end of the term examination were compared for the ‘Module with Socrative’ versus the ‘Modules without Socrative’ MCQs (Table 5). As illustrated, a significantly higher number of students among the entire class and high scorers answered correctly in the ‘Module with Socrative’ compared to the ‘Modules without Socrative’ [whole class performance (% of students who answered correctly): 83.53 ± 12.98 vs 71.50 ± 21.84, p < 0.05; high scorers: 92.99 ± 9.25 vs 84.77 ± 20.86, p < 0.05]. Among low scorers, the difference was statistically insignificant, although their performance was better in MCQs from lectures with Socrative activity and the p-value was close to the significance level (p = 0.06).

Table 5.

Students’ Performance in Examination Out of Modules With/or Without Socrative.

No.	Student categories	% of the students who answered correctly		p-value
		Module with Socrative	Module without Socrative
1.	All students	83.53 ± 12.98	71.5 ± 21.84	0.012
2.	High scorers	92.99 ± 9.25	84.77 ± 20.86	0.021
3.	Low scorers	66.3 ± 5.63	54.69 ± 7.82	0.06

When asked about the most suitable format for Socrative quizzes, the majority (99%) of study participants chose MCQs (Figure 1).

Figure 1.

Students' Preferences About the Format of Questions in Socrative.

Discussion

Most study participants enjoyed using the Socrative application on a mobile device. This positive attitude of students towards a mobile-based SRS is in complete agreement with Voelkel and Bennett (2014).²⁴ Students felt more actively engaged, encouraged, motivated, and interested in the lecture with the Socrative application. Further, it helped them focus more on key knowledge. These results are also in complete agreement with the studies reporting improved student interaction, engagement, and attention with the use of ordinary SRSs such as clickers.^17,25,26 Above all, students agreed that the mobile-based Socrative application stimulated student-instructor and student–student discussions. Our findings mirror a previous study in which Socrative was compared with an ordinary SRS. Students reported significantly higher active participation and a better environment for Socrative compared to a traditional SRS.²⁷ Our findings are in complete agreement with Abdallah,¹⁶ who reported that 85% of students agreed that using Socrative in lectures increased their participation and enhanced their understanding.

The comparison of male and female opinions on the use of Socrative revealed that differences between the two genders were insignificant (Table 2). Students' responses to all items in the survey were similar, regardless of the gender. Both groups equally enjoyed using Socrative, revealing that gender did not play a role in how students perceived the usefulness of learning with Socrative. These results align with the argument proposed by Yu (2015)²⁸ who found no statistically significant difference between male and female students’ perceptions of the use of SRSs in terms of satisfaction, perceived benefits, etc. Our results contradict a study in which females had significantly higher positive perceptions of clickers than males.²⁹

To determine whether students' positive attitudes toward Socrative were associated with their screen time, students were asked about their daily average screen time (for both academic and non-academic purposes like playing video games, watching movies, using social media and internet surfing). Pearson correlation did not reveal any significant association between total daily screen time and individual item scores (Table 3), which signifies that students' perception about the mobile-based Socrative SRS was not a function of their screen time. Had students’ attitude toward Socrative been related to their total screen time, female participants in our study would have had a significantly higher preference for Socrative compared to males (because females had significantly increased screen time than males; data not shown.) However, this was not found. All students enjoyed the Socrative experience and reported positive experiences, regardless of their total screen time.

Instructors' feedback is shown in Table 4. In addition to the points mentioned in Table 4, interviews with the instructors disclosed that Socrative was user-friendly, and there were no issues with registration, login, quiz-making, quiz-solving, and analysis of scores. Instructors believed that the ‘Instant Feedback’ feature of Socrative was priceless, as it displayed the students' responses the moment they submitted. When most students chose the incorrect answer, instructors were able to explain the correct one. Moreover, by analysing students' responses, instructors assessed the degree of understanding, provoked discussion, and tailored the subsequent lectures accordingly. As described by the instructors, disadvantages of using Socrative are: (1) a significant increase in session preparation time because the question-making process is time-consuming; (2) the use of Socrative during a lecture shortens the time used for the delivery of contents. This agrees with Knight and Wood,³⁰ who reported that use of SRSs requires time, which was previously used for delivery of the lesson's content.

There was significant improvement (p = 0.012) in the entire class performance in MCQs related to ‘Module with Socrative’ compared to performance in ‘Modules without Socrative’ (Table 5). Furthermore, higher scorers performed significantly better in ‘Module with Socrative’ than ‘Modules without Socrative’ (p = 0.021) in the exam. This agrees with Abdulla,¹⁶ who reported significantly enhanced performance of medical students in exam questions related to physiology concepts taught with a Socrative exercise, compared to those taught without Socrative. Among low scorers, though students' performance was higher in ‘module with Socrative’, statistical significance was not achieved. Improved exam performance in Socrative module could be due to the combined effects of (a) retrieval practice that led to better retention and recall; (b) Socrative quizzes acting as a ‘mock test’ in which students were better oriented toward the structure of final tests, although none of the MCQs from Socrative class quizzes were repeated in the end-of-term exam; (c) Socrative quizzes acting as ‘formative assessment’ to evaluate what the students have learned through the lectures, which helped identify and clarify the misconceptions among the students regarding lecture contents.

Our study has a limitation. Since the instructors surveyed their own students, this might entail bias.

Conclusion

Our study concludes that the use of Socrative in classrooms through smartphones is reliable and easy, and feedback from both students and instructors is positive. Moreover, it enhances students’ performance.

Recommendations

In view of our study results, instructors in every course should be encouraged to incorporate mobile-based applications in their lectures. It is probable that such diversification in the delivery mode will make students’ learning more active, effective, and enjoyable, without causing a spending increase for the universities.

Source of funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Conflict of interest

There is no conflict of interest.

Ethical approval

All procedures were in accordance with the ethical standards of the institutional Review Board of Imam Abdulrahman Bin Faisal University and with the 1964 Helsinki declaration and its later amendments.

Authors contributions

AAS conceived and designed the study, conducted research, collected & organized data. RL analyzed and interpreted data, and wrote initial & final draft of article. All authors have critically reviewed and approved the final draft and are responsible for the content and similarity index of the manuscript.

Acknowledgment

The authors fully acknowledge all the students of the third-year Medicine class at our university who participated in this research.