Skip to main content
PLOS ONE logoLink to PLOS ONE
. 2021 Sep 1;16(9):e0256814. doi: 10.1371/journal.pone.0256814

Effectiveness of blended learning in pharmacy education: An experimental study using clinical research modules

Balakrishnan Athira 1, Nair Sreedharan 1, Kunhikatta Vijayanarayana 1, Rashid Muhammed 1, Unnikrishnan M K 2,3, Jagannatha P S 4, Chandran Viji P 1, Khera Kanav 1, Thunga Girish 1,*
Editor: Ritesh G Menezes5
PMCID: PMC8409684  PMID: 34469484

Abstract

Background &objectives

Though there are studies to evaluate the effectiveness of blended learning in pharmacy education, most of them originate from USA and have used previous year students’ scores as control. Also there is less research in comparing use of self -regulated learning strategies between blended and other learning strategies. Primary aim was to evaluate the effectiveness of blended learning on knowledge score using clinical research modules. Secondary objective was designed to compare the use of self-regulated learning strategies between blended learning, web-based e-learning and didactic teaching.

Materials and methods

A prospective cluster randomized trial was conducted with didactic teaching as control and web-based e-learning and blended learning as interventions. The target population was final year Pharm D students. Outcome was assessed using a validated knowledge questionnaire, a motivated strategies for learning questionnaire and a feedback form. All statistical analyses were carried out using Statistical Package for Social Science (SPSS) Version 20.

Results

A total of 241 students from 12 colleges completed the study. Mean knowledge score of students in blended learning group was higher than those in the didactic teaching and web- based e- learning program (64.26±18.19 Vs 56.65±8.73 Vs 52.11±22.06,p<0.001).Frequency of use of learning strategies namely rehearsal, elaboration, organization and critical thinking was statistically significantly higher in the blended learning group compared to those of didactic and web-based e-learning group (p<0.05) But there were no statistically significant difference of motivational orientations between didactic and blended learning group except strategies of extrinsic goal orientation and self-efficacy. Students preferred blended learning (86.5%) over didactic and web-based e-learning.

Conclusion

Blended learning approach is an effective way to teach clinical research module. Students of blended learning group employed all motivational and learning strategies more often than students of the didactic and web- based e-learning groups except strategies of intrinsic goal orientation, task value, control of learning belief and help seeking.

Introduction

Blended learning (BL) or the integration of face-to-face and online instruction is widely adopted across the globe for tertiary education with the increasing use of information technology and electronic learning [1]. There is considerable growth in research analyzing the effectiveness of BL and understanding students’ preference and satisfaction towards BL in pharmacy education [26]. Some studies have concluded that BL is as effective as traditional teaching [2,3] while a few have reported that BL is more effective than conventional teaching [46]. However, most of these studies originate from US, and have employed previous year students’ scores as control. Moreover, only few studies come from developing countries. Currently lacking is, prospective controlled trials to analyze the effectiveness of BL in pharmacy education. A systematic review and meta-analysis on this topic is published by same authors [7]. Accordingly, we designed our study to evaluate the effectiveness of BL in final year Pharm D (Doctor of Pharmacy) students employing a 9 hour learning module entitled “Fundamentals of clinical research”. “Clinical research” was chosen because India is the preferred destination for multinational companies for clinical trials which raises the demand for skilled qualified personnel. Academia is unable to keep pace with industry because of quick regulatory reforms and technology upgrades thereby making timely syllabus revisions difficult. Training in these areas will help graduates gain more updated knowledge [8,9].

Self-regulated learning (SRL) involves cognitive, metacognitive, motivational, emotional, and behavioural aspects of learning and is currently a subject of extensive research and discussion in educational psychology. Student learning strategies are dynamic and developed to achieve learning objectives in a limited time span [10]. However, there is a little or no study comparing SRL strategies between BL and conventional methods [11]. In this study we have used Pintrich’ motivated strategies for learning questionnaire (MSLQ) [12,13] comprised of 15 variables and measures cognitive, metacognitive, motivational, and emotional aspects of learning procedures (Details of MSLQ is provided in S1 File). MSLQ successfully measures motivation and learning strategies in higher education, irrespective of discipline. Accordingly, we designed our secondary objective to compare the use of motivation and learning strategies between BL, web-based e-learning (WEL) and didactic teaching (DT). Blended learning refers to combination of online and face-to face, whereas web-based e-learning refers to all education that takes place online, while didactic teaching refers to all learning experiences that take place face-to-face in the classroom [1417].

Materials and methods

Design and implementation of web-based e-learning platform

Description of the learning module

This learning module was intended to teach site-based clinical trial activities to final year Pharm D students. The learning module comprised of five chapters, which included how to start a clinical trial, randomization, informed consent procedure, electronic case report forms, and adverse drug reaction reporting. Each chapter described the theory with the aid of case studies. Details of each chapter are outlined in Table 1. Experts from relevant areas verified contents in each chapter.

Table 1. Contents of learning module.
Chapter 1: How to start a clinical trial 1. Introduction
2. Confidentiality disclosure agreement, feasibility questionnaire, site evaluation
3. Investigators undertaking, financial disclosure agreement and clinical trial agreements
4. Site initiation
5. IRB-characteristics, composition &reviews
6. Reports to be submitted for IRB
Chapter 2: Recruitment, Screening &Randomization 1. Recruitment
2. Screening &randomization
3. Masking
Chapter 3: Informed consent process 1. Informed consent-general consideration
2. How to obtain informed consent
Chapter 4: Case report forms 1.Case report forms
2. Paper CRF and eCRF
3. How to capture data into eCRF
Chapter 5: Reporting of ADR 1. ADR-common terminologies
2. How to report ADR by Principal investigator
3. Responsibilities of stakeholders in ADR reporting
4. Compensation

Trial design

A prospective cluster randomized trial, having three arms with DT as control and WEL and BL as interventions 1 & 2 respectively, was conducted between October 2019 to February 2020.

Ethical consideration

Ethical approval for the study was obtained from Institutional Ethics Committee of Kasturba Hospital Manipal (25/2018), Karnataka, India. This study was registered under clinical trial registry of India (CTRI/2019/11/022021).

Inclusion criteria for colleges and participants

Colleges affiliated with All India Council for Technical Education (AICTE)/Pharmacy Council of India (PCI) were included. The target population was final year Pharm D students. Students who were interested in participating in the research study were included.

Recruitment of colleges and participants

Principal investigator approached institutional head through email providing a summary of the study, including randomization, along with an assurance of participant and institutional confidentiality. Details of study was summarized in each email and had informed that colleges would be allocated to control/interventions based on randomization. Consent was obtained from institutional heads for colleges’ participation.

Once consent obtained from institutional heads, investigators visited respective colleges and explained the study to participants. Participant information sheet(PIS) was provided to participants and all interested candidates willfully signed written informed consent form.

Sample size calculation

Sample size was calculated based on the primary outcome, total knowledge score. Effect size was calculated based on a pilot study, involving 22 students from Manipal College of Pharmaceutical Sciences, Manipal (S2 Appendix for pilot study details). Difference between the largest and smallest means divided by the square root of the mean square error, was used to compute the effect size [18]. A similar effect size was obtained by our meta-analysis of effectiveness of blended learning in pharmacy education [7]. Effect size (σ) of 0.6, at 5% level of significance, 80% power and design effect (d) of 2, the required sample size was 90 in each method of learning strategies. The required number of colleges in each geographical area was 3, assuming approx. 30 students in each class to each method of learning strategies. Students who participated in pilot study were excluded from the main study.

Randomization

Colleges that granted permission to conduct the study were grouped based on the geographical areas (GA) namely, Southern Karnataka (GA1), South East Karnataka (GA2), and Kerala (GA3) First, these geographical areas were randomly allocated to three methods of learning strategies and within each geographical area, 4 colleges were selected. There was no random selection of institutions in GA1 as there was only 4 institutions agreed to participate. Eight institutions out of ten agreed to participate were chosen at random in GA2 and GA3 (To reduce the contamination bias, we chose geographical areas as clusters. If students from surrounding colleges enrolled for different arms, there is a chance that they will share their login id & password).

Study procedure

DT, WEL and BL had the same course contents. All face- to- face sessions in DT & BL were conducted in respective colleges by same experts. Chapters were presented in the format of Microsoft PowerPoint with synchronous voice over slides in website. Script of voice was also attached in platform. Same PowerPoint slides were used to take classes in DT. Website contained 5 chapters, each lasting one hour as well as 4 hours of additional case studies and simulated forms, totaling 9 hours. Each chapter contained theoretical explanation, followed by reinforcing the concept with one or two case studies. Additional case studies and simulated forms were attached to the platform, which was available only to students of WEL. There was a case scenario, based on which students were asked to fill the simulated forms to provide realistic experiences. Simulated form included feasibility questionnaire, ADR reporting form, electronic case report form, and informed consent form. Students of BL were unable to access additional case studies and simulated forms on the website which was taught by face-to-face instruction. Students were required to sign up for first time and log in to the web page to access the learning materials. Students were enrolled for a study period of 8 weeks (details of each learning strategy is depicted in S3 Appendix).

Web-based e-learning platform for Fundamentals of clinical research

CLINIC E-LEARNING (http://clinilearn.in/my/): A new learning management system, CLINILC E-LEARNING, was developed to implement training program for students. This platform was developed using Moodle platform Version 3.2. PHP 7.0x was used to code the contents. CLINIC E-LEARNING contained five chapters as described above, and each chapter had pre-and post-test, case study questions, case study for references, important links and files, simulated forms, assignments and feedback forms. Only eligible participants were permitted to access the learning management system through login id and password. (See S4 Appendix for details of Website).

Experimental and control group

DT-(control)

Students attended the 9 hour face-to-face lecture delivered by experts with the aid of PowerPoint slides (Same slides of website). Face-to face session was conducted via workshops in each selected college.

WEL-(Experimental group 1)

WEL group were provided with URL and were asked to sign up with the option to access the website anytime, anywhere. They could clarify doubts by posting their queries to experts in the learner management system. Students were told to complete 9 hour learning module (5 chapters within 6 weeks and remaining two weeks for case study analysis and practicing simulated forms) within 2 months.

BL-(Experimental group 2)

BL students were provided with 5-hour e-learning and 4 hour face-to-face instruction, totaling 9hours. Students of BL studied the same audiovisual slides (in the same website) during the same period. Followed by, 4 hour discussion by experts (face-to-face instruction provided by the DT team) on additional case studies and important forms used in clinical trials.

Outcome assessment

We employed the same tests to assess outcomes in all three groups. Outcomes were measured immediately after the completion of the learning module. Outcome assessment consisted of 3 parts (see S5 Appendix for questionnaires). All questionnaires were validated by 5 experts for content validity.

  1. Evaluation of knowledge gain: Pre-test and post-test, consisted of same multiple choice questions (50 questions, ten questions from each chapter; 1 mark for each correct answer).

    Evaluation of conceptual understanding: Case study questions (had 17 items 2 marks for each correct answer).

  2. Evaluation of usage of SRL: Motivated strategies for learning questionnaire (MSLQ) by Pintrich et al, comprised of 81 questions. Scoring of MSLQ questionnaire is provided in appendix A.

  3. Evaluation of student’s experience and satisfaction: Feedback collected in two sessions with 12 questions.

Statistical analysis

All statistical analyses were carried out using Statistical Package for Social Science (SPSS) Version 20. Continuous variables were reported in terms of mean and standard deviation whereas categorical variables were expressed as percentages. Shapiro Wilks test was performed to ensure the normal distribution of variables. One way Anova or Kruskal Wallis test was conducted to find significant differences in test scores between the groups. Pairwise comparison performed by Dunn’s post hoc test, and Games Howell test. Relationships between variables were examined by multiple linear regressions. Spearman correlation was conducted to find out the correlation between variables. Mann Whitney U test performed to analyze difference in opinions of students of WEL and BL groups.

Results

A total of 317 students enrolled from 12 colleges, of them 241 students completed the learning module on Fundamentals of clinical research. Only students who finished both pre-test and post-test were included for the analysis.

Comparison of knowledge score

Pretest scores were compared by one way ANOVA and found that there was no significant difference between groups, p>0.05. As shown in Table 2, mean difference between post-test and pre-test, case studies score, and total score in BL was higher than DT and WEL. There was statistically significant difference in mean difference score (Post to pretest) in knowledge (p<0.001), post-case study score (p = 0.034), and total score (p<0.001) between learning strategies. BL showed statistically significant highest total score than didactic teaching (p<0.001) and web-based e-learning program (p<0.001) by Dunn’s post hoc test. Results are presented in Table 2.

Table 2. Comparison of knowledge score between learning strategies.

Study Groups N Age Gender (Male: Female) Pretest (Mean ±SD) Posttest (Mean ±SD) Mean Difference (Mean ±SD) Post case study score (Mean± SD) Total score
1. Control (Didactic teaching) 87 22.43±0.58 13:74 21.82± 5.80 33.50±6.63 11.67 ± 6.37 23.14±5.95 56.65±8.73
2. Experimental Group 1(Web -based e-learning) 62 22.61±0.49 14: 48 23.54±11.04 30.59±14.12 7.04± 8.00 21.51±9.80 52.11±22.06
3. Experimental Group 2(Blended learning) 92 22.51±0.54 9:83 23.69± 8.82 39.39±11.02 15.69± 9.88 24.87±8.49 64.26±18.19
F value 5.30 4.94 1.24 30.50 20.33 6.74 21.38
p value 0.07 0.084 0.289 <0.001 <0.001 0.034 <0.001
Post hoc test
 1. DT Vs WEL
 2.WEL Vs BL
 3. DT Vs BL
1.0.001
2.<0.001
3.0.004
1.0.290
2.0.047
3. 0.165
1. 0.911
2. <0.001
3<0.001

Pretest and mean difference was compared using ANOVA. Post hoc test by Games Howell test as equal variance is not assumed. Kruskall Wallis test for not normally distributed variable (Age, Posttest, Post case study score, total score) and pairwise comparison by Dunn’s post hoc test. Gender compared by chi-square test.

Multiple linear regression model showed an R2 value of 0.41, which means independent variable (pretest, gender and learning strategies keeping BL as reference category) can explain 41% of the variability of dependent variable (total score). The coefficient on Intervention (β3 = -5.32) means that, holding all other variables constant, the score on the final test of a student who was taught via the DT is expected to decrease 5.32 points compared to a student who received the BL intervention. Similarly, total score of WEL group students are expected to decrease -12.03 compared to BL group. The findings are depicted in Table 3.

Table 3. Multiple linear regression demonstrating the relationship of total score with other variables.

Model variable Coefficient Estimate Standard Error P value
Intercept(β0) 36.51 3.67 P<0.01
Pretest(β1) 1.18 0.01 P<0.01
Gender(β2) -0.43 2.41 0.858
Didactic teaching(DT) (β3) -5.32 2.00 0.007
Web-based e-learning(WEL) (β4) -12.03 2.21 P<0.01

Multiple linear regression with total score as dependent variable. For learning strategies, BL kept as reference level.

Comparison of MSLQ scores

It was found that usage of SRL strategies were higher in BL group than DT and WEL except intrinsic goal orientation, task value, control of learning belief and help seeking. There was no statistically significant difference in test anxiety, time management and effort regulation between groups (p>0.05). SRL strategies are composed of 15 variables and were analyzed by Kruskal Wallis test and presented in Table 4.

Table 4. Comparison of MSLQ scores between learning strategies.

MSLQ Domain Teaching method Mean ±S.D P value Pairwise comparison
 1. DT Vs WEL
 2. WEL Vs BL
 3. DT Vs BL
Intrinsic goal orientation Didactic 5.42±0.92 0.016 1.p = 0.026
2.p = 0.038
3.p = 0.905
Web based 5.00. ± 1.11
Blended 5.39± 0.95
Extrinsic goal orientation Didactic 4.49± 1.38 0.010 1.p = 0.754
2.p = 0.068
3.p = 0.015
Web based 4.72± 1.09
Blended 5.08 ±1.19
Task value Didactic 5.41 ± 0.90 0.010 1.p = 0.017
2.p = 0.025
3.p = 1.00
Web based 4.99 ± 1.06
Blended 5.38 ±1.06
Control of learning belief Didactic 5.47± 0.88 P<0.01 1.p < .001
2.p = 0.001
3.p = 1
Web based 4.84. ± 0.89
Blended 5.39 ±0.92
Self-efficacy Didactic 4.77 ± 0.87 P<0.01 1.p = 1
2.p = 0.002
3.p<0.001
Web based 4.84 ± 0.93
Blended 5.31± 0.97
Test anxiety Didactic 4.06± 1.35 0.059
Web based 4.38 ±0.97
Blended 4.53± 1.34
Rehearsal Didactic 4.61± 1.02 P<0.01 1.p = 1
2.p = 0.001
3.p = 0.002
Web based 4.60 ± 1.04
Blended 5.09±1.07
Elaboration Didactic 4.79 ±1.14 P<0.01 1.p = 0.495
2.p<0.001
3.p = 0.015
Web based 4.69 ±0.90
Blended 5.22 ±1.00
Organization Didactic 4.72 ± 1.18 P<0.01 1.p = 0.237
2.p<0.001
3.p = 0.040
Web based 4.50 ± 1.08
Blended 5.14 ±1.08
Critical thinking Didactic 4.71 ±0.99 0.002 1.p = 1
2.p = 0.003
3.p = 0.032
Web based 4.66 ± 1.01
Blended 5.10± 1.08
Metacognition Didactic 4.67 ±0.76 P<0.01 1.p = 0.029
2.p<0.001
3.p = 0.164
Web based 4.43 ± 0.59
Blended 4.89± 0. 77
Time Didactic 4.32± 0.68 0.077
Web based 4.25 ± 0.47
Blended 4.43± 0. 50
Effort regulation Didactic 4.22± 1.05 0.585
Web based 4.06± 0.52
Blended 4.25 ± 0.70
Peer learning Didactic 4.84 ±1.28 0.005 1.p = 0.016
2.p = 0.007
3.p = 1
Web based 4.40 ±1.18
Blended 4.93 ± 1.08
Help seeking Didactic 4.6 ± 1.00 0.003 1. p = 0.002
2.p = 0.091
3.p = 0.507
Web based 4.20± 0.79
Blended 4.40 ± 0.84

P value calculated using Kruskal Wallis test and pairwise comparison by Dunn’s post hoc test.

Further pairwise comparison between the groups showed that there was no statistically significant difference in usage of intrinsic goal orientation, task value and control of learning belief between DT and BL (p>0.05). Pairwise comparison of usage of rehearsal, elaboration, organization, and critical thinking indicated there was no statistically significant difference between the DT and WEL. Peer learning showed no significant difference between DT and BL, whereas significant difference exist between the DT and WEL in help seeking.

To determine the relationship between motivational and learning strategy use and final score, Spearman correlation was performed separately for DT, WEL and BL groups. Result showed final score had significant weak positive correlation with extrinsic goal orientation (r = 0.213), control of learning belief (r = 0.280) and rehearsal(r = 226) for students of BL. We did not find any significant correlation between final score and MSLQ strategies in DT and WEL.

Comparison of students’ opinion about contents of web-based e- learning platform

It was found that there was no statistically significant difference(p>0.05) in opinions regarding design of module, clarity of explanation, comprehensive coverage of subject matter, and relevancy of hyperlinks between the WEL and BL groups who used the website for learning. Students of BL were asked their preference for learning strategies and found that 86.5%of students prefer BL, 2.7% and 9.5% preferred DT and WEL, respectively. Students responded that they were satisfied with the clinical research modules and accomplished various learning activities. Mann-Whitney test showed no significant difference between students’ opinion (p>0.05).

A total of 92.2% students enrolled for didactic teaching responded that they were satisfied with workshops. More than three quarters of DT students agreed that they were satisfied with clarity of explanation and discussion of module lesson, comprehensive coverage of subject matter, consistency of content with subject objectives and syllabus Results are presented in Table 5.

Table 5. Students’ attitude towards web -based e-learning program.

SL. NO: Questions Learning strategy excellent good fair Poor P value
1 Design of the modules Blended learning 37.9% 37.9% 22.7% 1.5% 0.252
Web-based 29.8% 38.3% 25.5 6.4%
2 Explanation of purpose, objectives, and grading procedures Blended learning 47% 39.4% 10.6% 3% 0.874
Web-based 51.1% 27.7% 8.5% 12.8%
3 Clarity of explanation and discussion of the module lessons. Blended learning 40.9% 42.4% 9.1% 7.6% 0.745
Web-based 42.6% 31.9% 17% 8.5%
4 Consistency of content with subject objectives and syllabus Blended learning 33.3% 47% 18.2% 1.5% 0.908
Web-based 38.3% 36.2% 17% 8.5%
5 Comprehensive coverage of subject matter Blended learning 43.9% 42.4% 12.1% 1.5% 0.061
Web-based 31.9% 40.4% 17% 10%
6 Relevance of hyperlinks (if any) Blended learning 36.4% 36.4% 16.7% 9.1% 0.210
Web-based 21.3% 42.6% 14.9% 12.8%

P value calculated using Mann-Whitney test.

Discussion

Published literature suggests that BL generates better performance in pharmacy education, but most of such studies used previous year students’ score as control [19,20]. To the best of our knowledge, this report will be the first attempt to evaluate the effectiveness of BL in clinical research modules in pharmacy education. Despite the fact that the term "blended learning" is commonly used, there are some uncertainty about what it means. These approaches are still little understood in higher education, and their descriptions in the literature are inconsistent. In higher education, the term “hybrid” is often used synonymously with “blended” learning, “flipped,” “online,” or “technology-enhanced” learning [15]. A recent article by Kendra Gagnon distinguish blended and hybrid learning based on face-to- face time [16]. Blended learning does not replace face-to-face time but hybrid learning does. To be more specific, hybrid learning reduces seat time in class. As a result, our model will be better suited to hybrid learning. However, we used the term blended learning because literatures describe hybrid learning is a type of blended learning, and a precise definition remains an ongoing conversation [14,15]. In comparison to blended learning, the term hybrid learning might have been more extensively used in practice than research as there are less highly cited articles on hybrid learning. According to Stefan Hrastinski it is important that researchers and practitioners carefully describe what blended learning means to them. He also proposes, researchers should carefully consider while using a more precise, descriptive term as a supplement or replacement for blended learning [15]. So we believe we have adequately addressed this issue.

Though Clinical trials appear to be concentrated in developed nations, highest percentage increase in registered human clinical research, particularly phase 3 trials are increasing in developing countries [21]. So training in these areas have the potential to enhance students’ basic understanding of how the industry works. We have shown that BL is more effective than DT and WEL in teaching clinical research modules, which is in agreement with previous reports in similar settings [22,23]. Ample data have shown that case studies are a gratifying and motivational educational tool that extends declarative and procedural knowledge/expertise [24]. We incorporated case studies in all three learning strategies. Both DT and BL used case study discussions as an active learning strategy in the classroom and enhanced student tutor interaction. Though case studies were incorporated on website and students could post their queries in learning management system itself, there were no provision of face–to-face tutor to student interaction for students who participated in WEL. Students who experienced face-to- face discussion (DT & BL) demonstrated better performance than WEL group, underlining the importance of face-to-face interaction in improving the final score.

In blended learning model class time was effectively used for active learning strategies as students had already viewed recorded videos. This allowed more meaningful face-to face interaction, application of course content and also provided an equal educational value as didactic teaching. BL students spent less time in class compared to DT, which echoes the result of a similar study by the US Department of Education on Distance Education as well as another study by Philips JA [25,26].

BL employed motivational and learning strategies more frequently than DT, except in strategies of intrinsic goal orientation, task value, control of learning belief and help seeking. WEL students scored less in all scales of motivational and learning strategies compared to BL. However, DT students employed all motivational and learning strategies more often than WEL students, except in extrinsic goal orientation, self-efficacy and test anxiety. Interestingly, there is no significant difference between DT and BL in the use of motivational strategies, except in extrinsic goal orientation and self-efficacy; suggesting that face-to—face interaction with tutor considerably influences the use of motivational strategies. This result is in accordance with a study by Kassab et al [27]. DT scored highest for intrinsic goal orientation but least for extrinsic goal orientation, possibly because DT students were personally motivated by the tutor.

BL showed statistically significant highest score in all scales of cognitive strategies (rehearsal, elaboration, organization, and critical thinking) than DT and WEL. Possible reason could be, in BL, students are getting more meaningful interaction with tutors as they are familiar with the content before class. This finding is mostly in accord with a study by Ruchan Uz [28] who reported use of blended instruction is more effective than traditional instruction in terms of developing learning strategies. However our results are in contrast to those of Broadbent [11] who reported that online learning students reported more frequent usage of learning strategies than BL.

Another key finding of this study was that no significant difference was found in metacognitive self-regulation score between DT and BL. So we assume that face-to-face interaction with tutors can demonstrate what to learn, when to learn and how to learn which in turn affect on metacognitive approaches such as planning, monitoring, and regulating.

As expected peer learning and help seeking score was less in WEL group. Help seeking was highest in DT. Understandably, online students felt more isolated when deprived of a sense of community. This is in line with previous studies [11,29].

We have shown that only extrinsic goal orientation, control of learning belief and rehearsal strategies had a weak positive correlation with final score of clinical research examination in BL. On the other hand, SRL did not correlate with final score in DT and WEL.

To the best of our knowledge, there was only one study by Broadbent et al [11] who compared the use of SRL between online and blended environment in higher education. Our result contradicts Broadbent, possibly because, we employed the same learning module for all evaluation unlike much more broad based subject range by Broadbent et al. Moreover, online students had an opportunity for one-to-one interaction with peers and instructors in Broadbent’s study. Since student’s attitude towards course content also determine SRL, Broadbent’s results may not apply to our situation.

In addition, the blended-learning approach used in this course was generally well received by the students. These findings align with other studies that report positive student perceptions of BL in pharmacy education [30,31].

There are some inherent limitations to this study. Firstly, as this study was focused on a single topic, our findings may not be extrapolated to other topics of pharmacy or any other healthcare disciplines. Second, this study accessed short term retention of material. Third, we did not assess how many times students logged in to the website and how long they spent to viewing the materials. As stated by previous reported study [26] we could overcome this by stating that students logging in does not necessarily mean they were actively studying the content. The time obtained may have provided time that the students may not have been physically seated in front of the screen watching the e-lectures because the lecture window would remain active as long as they were signed on to the web-based e-learning portal. Fourth, due to the anonymous nature of feedback form, we were not able to link students’ satisfaction level and academic achievement.

Conclusion and future research lines

Blended learning approach, with online and face-to-face instruction is an effective way to teach clinical research module. Results showed students of blended learning group employed all motivational and learning strategies more often than students of didactic and web- based e-learning groups except strategies of intrinsic goal orientation, task value, control of learning belief and help seeking. Also Students prefer blended learning program than didactic and pure online courses.

Blended learning will be the future of the education sector, since online learning cannot replace physical experience at universities. Given the rapid growth of online studies in last decades, and in the wake of COVID 19, it is paramount to evaluate the effectiveness of computer mediated instruction and to understand how students utilise SRL strategies to achieve academic success. There is an urgent need of fine grained studies with solid experimental design to evaluate the use of SRL between various pedagogies. More insights are needed to correlate the use of SRL and academic achievement. Most BL studies, including current research have assessed short-term retention of knowledge, so future research should address long-term retention of knowledge.

Supporting information

S1 Appendix. Motivated strategies for learning questionnaire (MSLQ).

(DOCX)

S2 Appendix. Pilot study details.

(DOCX)

S3 Appendix. Various strategies used for different teaching learning method.

(DOCX)

S4 Appendix. Web-based e-learning program.

(DOCX)

S5 Appendix. Questionnaires.

(DOCX)

S1 File. Students demographics, knowledge score & MSLQ score.

(XLSX)

S2 File. Feedback file.

(XLSX)

Acknowledgments

Authors also would like to acknowledge Department of Pharmacy Practice, Manipal College of Pharmaceutical Sciences, and Manipal Academy of Higher Education for all the support and facilities for the best possible completion of this study. We also would like to acknowledge Muscle Infotech. Bangalore for the website development.

Data Availability

All relevant data are within the manuscript and its Supporting Information files.

Funding Statement

The authors received no specific funding for this work.

References

  • 1.Graham CR. Emerging practice and research in blended learning. Handbook of distance education. 2013; pp.333–50. [Google Scholar]
  • 2.Wilson J.A., Waghel R.C. & Dinkins M.M. Flipped classroom versus a didactic method with active learning in a modified team-based learning self-care pharmacotherapy course. Curr Pharm Teach Learn. 2019; 11(12); 1287–1295. doi: 10.1016/j.cptl.2019.09.017 [DOI] [PubMed] [Google Scholar]
  • 3.Nazar H., Omer U., Nazar Z. & Husband A. A study to investigate the impact of a blended learning teaching approach to teach pharmacy law. Int J Pharm Pract. 2019; 27(3): 303–310. doi: 10.1111/ijpp.12503 [DOI] [PubMed] [Google Scholar]
  • 4.He Y, Lu J, Huang H, He S, Ma N, Sha Z, et al. The effects of flipped classrooms on undergraduate pharmaceutical marketing learning: A clustered randomized controlled study. PLoS One.2019; 14(4):e0214624. doi: 10.1371/journal.pone.0214624 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Newsom L, Proctor R, Marshall LL, Liao TV. Implementation and evaluation of problem-based video podcasts in an introductory pharmacokinetics course. Curr Pharm Teach Learn. 2019; 11(12):1213–20. doi: 10.1016/j.cptl.2019.09.003 [DOI] [PubMed] [Google Scholar]
  • 6.Goh C.F. & Ong E.T. Flipped classroom as an effective approach in enhancing student learning of a pharmacy course with a historically low student pass rate. Curr Pharm Teach Learn. 2019; 11(6), 621–629. doi: 10.1016/j.cptl.2019.02.025 [DOI] [PubMed] [Google Scholar]
  • 7.Balakrishnan A, Puthean S, Satheesh G, M. K. U, Rashid M, Nair S, et al. Effectiveness of blended learning in pharmacy education: A systematic review and meta-analysis. Plos One.2021;16(6): e0252461. doi: 10.1371/journal.pone.0252461 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Booming Clinical Trial Market in India. RNCOS online business research.2007.
  • 9.Balakrishnan A, Thunga G, Sreedharan Vijayanarayana K, Khera K. Bridging the Gap Between Industry and Academia in Pharmaceutical Education. Ind.J. Pharm.Edu.Res. 2017; 51(2S):S46–S53. [Google Scholar]
  • 10.Panadero E. A review of self-regulated learning: Six models and four directions for research. Front Psychol. 2017; 8:422. doi: 10.3389/fpsyg.2017.00422 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Broadbent J. Comparing online and blended learner’s self-regulated learning strategies and academic performance. Internet High Educ. 2017; 33: 24–32. [Google Scholar]
  • 12.Pintrich P R, de Groot EV. Motivational and self regulated learning components of classroom academic performance. J Educ Psychol. 1990; 82(1):33–40. [Google Scholar]
  • 13.Pintrich PR, Smith DAF, Garica T, McKeachie WJ. A manual for the use of motivated strategies for learning questionnaire(MSLQ). Ann Arbor: University of Michigan.1991. [Google Scholar]
  • 14.Anthony B., Kamaludin A., Romli A. et al. Blended Learning Adoption and Implementation in Higher Education: A Theoretical and Systematic Review. Tech Know Learn. 2020. 10.1007/s10758-020-09477-z. [DOI] [Google Scholar]
  • 15.Hrastinski S. What Do We Mean by Blended Learning?. TechTrends. 2019; 63, 564–569. 10.1007/s11528-019-00375-5. [DOI] [Google Scholar]
  • 16.Gagnon K, Young B, Bachman T, Longbottom T, Severin R, Walker MJ. Doctor of Physical Therapy Education in a Hybrid Learning Environment: Reimagining the Possibilities and Navigating a “New Normal”. Phys Ther. 2020;100(8):1268–77. doi: 10.1093/ptj/pzaa096 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Saichaie K. Blended, Flipped, and Hybrid Learning: Definitions, Developments, and Directions. New Directions for Teaching and Learning. 2020;2020(164):95–104. [Google Scholar]
  • 18.Introduction to SAS. UCLA: Statistical Consulting Group. from https://stats.idre.ucla.edu/sas/modules/sas-learning-moduleintroduction-to-the-features-of-sas/ (accessed June 2021).
  • 19.Liu Q., Peng W., Zhang F., Hu R., Li Y. & Yan W. The effectiveness of blended learning in health professions: systematic review and meta-analysis. J Med Internet Res. 2016;18(1), e2. doi: 10.2196/jmir.4807 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Gillette C., Rudolph M., Kimble C., Rockich-Winston N., Smith L. & Broedel-Zaugg K. A meta-analysis of outcomes comparing flipped classroom and lecture. Am J Pharm Educ. 2018; 82(5). doi: 10.5688/ajpe6898 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Drain PK, Parker RA, Robine M, Holmes KK. Global migration of clinical research during the era of trial registration. PLoS One. 2018;13(2):e0192413. doi: 10.1371/journal.pone.0192413 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.McLaughlin J.E. & Rhoney D.H. Comparison of an interactive e-learning preparatory tool and a conventional downloadable handout used within a flipped neurologic pharmacotherapy lecture. Curr Pharm Teach Learn. 2015; 7(1);12–19. [Google Scholar]
  • 23.Anderson H.G., Frazier L., Anderson S.L., Stanton R., Gillette C., Broedel-Zaugg K., et al. Comparison of pharmaceutical calculations learning outcomes achieved within a traditional lecture or flipped classroom andragogy. Am J Pharm Educ. 2017; 81(4). doi: 10.5688/ajpe81470 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Turk B, Ertl S, Wong G, Wadowski PP, Löffler-Stastka H. Does case-based blended-learning expedite the transfer of declarative knowledge to procedural knowledge in practice? BMC Med Educ. 2019; 19(1):1–0. doi: 10.1186/s12909-018-1418-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Parsad B, Lewis L, Tice P. Distance education at degree-granting postsecondary institutions: 2006–2007. Washington, DC: National Center for Education Statistics, Institute of Education Sciences, US Department of Education; 2008. Retrieved from http://nces.ed.gov/pubs2009/2009044.pdf. [Google Scholar]
  • 26.Phillips J.A., Schumacher C. & Arif S. Time spent, workload, and student and faculty perceptions in a blended learning environment. Am J Pharm Educ. 2016; 80(6). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Kassab SE, Al-Shafei AI, Salem AH, Otoom S. Relationships between the quality of blended learning experience, self-regulated learning, and academic achievement of medical students: a path analysis. Adv Med Educ Pract. 2015; 6:27. doi: 10.2147/AMEP.S75830 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Uz R, Uzun A. The Influence of Blended Learning Environment on Self-Regulated and Self-Directed Learning Skills of Learners. Eur. J. Educ. Res. 2018; 7(4):877–86. [Google Scholar]
  • 29.Chen C.S. Self-regulated learning strategies and achievement in an introduction to information systems course. Information technology, learning, and performance journal. 2002; 20(1), 11–25. [Google Scholar]
  • 30.Gloudeman M.W., Shah-Manek B., Wong T.H., Vo C. & Ip E.J. Use of condensed videos in a flipped classroom for pharmaceutical calculations: Student perceptions and academic performance. Curr Pharm Teach Learn. 2018; 10(2):206–210. doi: 10.1016/j.cptl.2017.10.001 [DOI] [PubMed] [Google Scholar]
  • 31.Prescott W.A., Woodruff A., Prescott G.M., Albanese N., Bernhardi C. & Doloresco F. Introduction and assessment of a blended-learning model to teach patient assessment in a doctor of pharmacy program. Am J Pharm Educ. 2016; 80(10). [DOI] [PMC free article] [PubMed] [Google Scholar]

Decision Letter 0

Ritesh G Menezes

4 May 2021

PONE-D-21-09098

Effectiveness of blended learning in pharmacy education: An experimental study using clinical research modules

PLOS ONE

Dear Dr. Thunga,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by 30-May-2021. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Prof. Ritesh G. Menezes, M.B.B.S., M.D., Diplomate N.B.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

Reviewer #4: Yes

Reviewer #5: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

Reviewer #4: Yes

Reviewer #5: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

Reviewer #4: No

Reviewer #5: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Thank you for the invitation to review this interesting manuscript. My comments are as follows:

Line 108: The spelling of the e-learning platform seems to be wrong.

Lines 119 and 120: Can the IEC of Kasturba Hospital provide ethical approval for a multi-centre study?

There are missing words and other issues with language in many areas. I would recommend copyediting of the manuscript for language and grammar. Punctuation and spacing between words may also need corrections.

The evaluation questionnaires seem to be long, and students may have to spend several minutes to complete the same. This could have been a problem. How much time did it take on an average to complete the questionnaires?

Line 208: The number of students who completed the module is only 76% of those who enrolled. Will this impact the study?

Line 290: The references should be cited as [19,20].

The authors have mentioned several learning strategies and other constructs in the article. A brief explanation of these will be helpful.

It is good that the authors have included different supporting material with the article.

Reviewer #2: After reviewing this manuscript, it became known that this needs some minor corrections before acceptance. Sentences must be reformed with clear elucidation and a more scientific approach. Also, references must be written in accordance to guidelines provided by the journal.

Reviewer #3: A. Is the manuscript technically sound, and do the data support the conclusions?

The research objectives are relevant. However, the following points needs to be addressed:

1. Regarding the study design, it was mentioned that the three interventions were randomly allocated to three geographical areas, and from where colleges were randomly selected. What is the rationale of having only three clusters? Why did the authors select the geographical area, instead of college, as the cluster level? Please provide the information on the number of colleges considered for the random selection of 12 colleges. It would be helpful for the readers if information available on cluster level factors that might influence the results.

2. Regarding sample size, how was the effect size calculated? on which outcome variable?

3. The study compares three types of learning approaches: BL, DT & WEL. In DT, whole 9 hours learning through face to face; in BL, the 5 hrs didactic part offered through online and 4 hrs case studies through face-to-face discussion; while in WEL, whole 9 hours through web based. However, it is rather unclear, what mentioned here as BL is a blended learning, or flipped classroom, or hybrid learning (for me, it’s more like hybrid learning!; authors used these approaches interchangeably though they are not the same)? All except one references in the introduction were about flipped classroom approach. Interpretation and discussion of results needs to be updated accordingly.

B. Has the statistical analysis been performed appropriately and rigorously?

4. Analysis performed as individually randomised trial!

5. For tables, please provide footnote on method of hypothesis testing

6. In table 2, provide the post-hoc comparison for all significant variables

7. Please clarify, what it means by 1, 2 & 3 in post hoc comparison

8. In table 3 regression modelling, how did the variable selection perform? Why only intervention 2 included?

C. Have the authors made all data underlying the findings in their manuscript fully available?

9. Only summary statistics were provided

D. Is the manuscript presented in an intelligible fashion and written in standard English?

10. Needed substantial revision due to typos (e.g. …website contains 5 chapters per hour…[ page 8]), unclear sentences, and grammatical mistakes.

E. Additional comments:

11. Introduction lacks literature review on the research problem, insufficient background information, and needs to be more structured. All except one reference in the introduction were about flipped classroom approach. The second part of both paragraphs in the introduction better fit in the methods section/discussion section. Objectives can be listed at the end.

12. In the methods section, better move the details of web implementation into under “study procedure” section.

13. Discussion needs to be more structured and include relevant references.

Reviewer #4: Thank you for the opportunity to review this paper. Generally, the topic would be of interest to many readers. Below are suggestions to address

1. Introduction - Why Pintrich’ motivated strategies for learning questionnaire (MSLQ) is selected?

2. Outcome Assessment: How did the items in pre-test and post-test generated? Evaluation of students’ experience and satisfaction – is it newly constructed items or adopted from previous study? Validation?

3. Comparison of knowledge score –

-Line 213 - “As shown in table3, mean difference between post-test and pre-test, case studies score, and total score in BL was higher than DT and WEL.”- is it Table 2 or Table 3?

- Case study scores – pre or post?

- Line 224-225 “Students’ pretest score, and whether they were in BL had a significant influence on final score.”- please clarity

4. The paper also includes inconsistencies e.g. decimal points, capital letter/small letter and would benefit from linguistic editing by a native English speaker prior to resubmission.

Reviewer #5: 1. The objective was to evaluate the effectiveness of blended learning on knowledge score using clinical research modules. Assessing knowledge on a specific subject to demonstrate increased competency in that particular field. However, in this study it was assessed only at the basic level, which are insufficient to argue for the value of learning. I would suggest to evaluate the behavior of the students using objective assessment, including direct observation, an interview or a skills demonstration.

2. The effectiveness of a blended learning environment can be assessed through analysing the relationship between student characteristics/background, design features and learning outcomes. It is always better to determine the significant predictors of effectiveness using learning outcomes as dependent variable.

3. The language is ambiguous and didn't reflect the context clearly.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Pathiyil Ravi Shankar

Reviewer #2: No

Reviewer #3: Yes: Royes

Reviewer #4: No

Reviewer #5: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachment

Submitted filename: PONE-D-21-09098_reviewer.docx

PLoS One. 2021 Sep 1;16(9):e0256814. doi: 10.1371/journal.pone.0256814.r003

Author response to Decision Letter 0


18 Jun 2021

Reviewer 1:

1. Line 108: The spelling of the e-learning platform seems to be wrong- Corrected.

2. Lines 119 and 120: Can the IEC of Kasturba Hospital provide ethical approval for a multi-centre study?- Kasturba hospital approved protocol and after submitting permission letter from each colleges, Ethics committee allowed to start the study. Study is conducted in different phases. Pilot study conducted in Manipal college of pharmaceutical sciences (IEC of Kasturba hospital affiliated) and appropriate revisions are made based on pilot study. These changes along with permission letter from each colleges again submitted to the IEC for the next phase approval.

3. There are missing words and other issues with language in many areas. I would recommend copyediting of the manuscript for language and grammar. Punctuation and spacing between words may also need corrections- Corrected

4. The evaluation questionnaires seem to be long, and students may have to spend several minutes to complete the same. This could have been a problem. How much time did it take on an average to complete the questionnaires?-

On average, 25 mts for MSLQ questionnaire, 1 hour 10 mts for knowledge test. Usually longer questionnaire will have lower response rate in survey. But its an interventional study & we have informed colleges &students about the nature of study. Only interested students from each colleges participated. So we don’t think number of questions will have an impact in our study.

5. Line 208: The number of students who completed the module is only 76% of those who enrolled. Will this impact the study?

Recommended sample size for three arms was 270, out of that 241 completed. So it may not affect the study. Major reason for dropout was students were absent on particular day of exam.

6. Line 290: The references should be cited as [19,20].-corrected

The authors have mentioned several learning strategies and other constructs in the article. A brief explanation of these will be helpful. –included, see S3 appendix for learning strategies & S1 for MSLQ variables.

It is good that the authors have included different supporting material with the article.

Reviewer 2:

After reviewing this manuscript, it became known that this needs some minor corrections before acceptance. Sentences must be reformed with clear elucidation and a more scientific approach. Also, references must be written in accordance to guidelines provided by the journal.

Manuscript edited and references corrected

Reviewer 3:

The research objectives are relevant. However, the following points needs to be addressed:

1. Regarding the study design, it was mentioned that the three interventions were randomly allocated to three geographical areas, and from where colleges were randomly selected. What is the rationale of having only three clusters? Why did the authors select the geographical area, instead of college, as the cluster level? Please provide the information on the number of colleges considered for the random selection of 12 colleges. It would be helpful for the readers if information available on cluster level factors that might influence the results.

Included these points in methodology.

We have selected geographical areas as cluster to minimize contamination bias. If students from nearby colleges enrolled for different arms, there are chances of exchanging login ID and Password. So that may affect the study. So we choose different geographical areas .

A total of 14 colleges agreed to participate the study from three GAs out of 25. In GA1, only four colleges decided to participate, and there was no random selection. Eight institutions were chosen at random from the ten colleges that provided approval for the study in GA2 and GA3

2. Regarding sample size, how was the effect size calculated? on which outcome variable?

Effect size was calculated based on total knowledge score. Effect size was calculated using difference between highest and lowest mean divided by pooled standard deviation). We obtained a value of 0.66 based on our pilot study. But pilot study results were not significant and conducted in small population.

So we conducted a meta-analysis of blended learning in pharmacy education, and found that pooled effect size of RCT 0.58.

Reference: Balakrishnan A, Puthean S, Satheesh G, M. K. U, Rashid M, Nair S, et al. Effectiveness of blended learning in pharmacy education: A systematic review and meta-analysis. Plos One.2021;16(6): e0252461.

So we choose 0.6 as final effect size (Not much difference between 0.58 and 0.6 as effect size, also we have taken 1 cluster extra to the sample size)

3. The study compares three types of learning approaches: BL, DT & WEL. In DT, whole 9 hours learning through face to face; in BL, the 5 hrs didactic part offered through online and 4 hrs case studies through face-to-face discussion; while in WEL, whole 9 hours through web based. However, it is rather unclear, what mentioned here as BL is a blended learning, or flipped classroom, or hybrid learning (for me, it’s more like hybrid learning!; authors used these approaches interchangeably though they are not the same)? All except one references in the introduction were about flipped classroom approach. Interpretation and discussion of results needs to be updated accordingly.

We agree with you based on definition” hybrid education is a type of blended learning that substitutes out-of-class online learning activities for face-to-face classroom time, means reducing face-face class room time”. But in higher education, the term “hybrid” is often used synonymously with “blended” learning or alongside terms such as “flipped,” “online,” or “technology-enhanced” learning. A recent systematic review on blended learning in higher education also mentioned “BL is frequently used with terms such as integrated, flexible, mixed mode, multi-mode or hybrid learning”

A recent article by Stefan Hrastinski, on “What Do We Mean by Blended Learning?” mentioned that “it is likely that papers that use more specific terms, such as blended synchronous learning or flipped classroom, can be found among the huge number of papers that fits under the inclusive blended learning umbrella”. So may be because of this most of our references from Introduction belongs to Flipped learning”. Also all these articles are mentioned their learning strategy as blended learning in introduction or discussion.

The search for a definition for blended learning has been productive, tough, and at times intimidating. The "hybrid learning" was commonly used before the term "blended learning" became popular. The terms blended learning and hybrid learning are often used interchangeably these days. (Graham 2009; Watson 2008). The term hybrid learning might have been more widely adopted in practice than in research, as there are quite few highly cited papers on hybrid learning, as compared with blended learning research. As far as our knowledge, no published literature clearly states difference between blended, flipped and hybrid learning. differences between BFH models remain ambiguous. Dziuban et al. (2018) suggest that blended is the “new normal” in course formats. Linder (2017) argues that the “flipped” model is simply a subset of the hybrid learning model. Cavanagh, Thompson, and Futch (2017) refer to hybrid as blended. The interchange of these terms persists across the literature.

As a result, there is a lot of ambiguity, and literature claims that hybrid learning falls within blended learning, we used the term blended learning. However we have included these points in discussion.

Included Discussion points:

Despite the fact that the term "blended learning" is commonly used, there are some uncertainty about what it means. These approaches are still little understood in higher education, and their descriptions in the literature are inconsistent. In higher education, the term “hybrid” is often used synonymously with “blended” learning, “flipped,” “online,” or “technology-enhanced” learning. A recent article by Kendra Gagnon distinguish blended and hybrid learning based on face to face time. Blended learning does not replace face to face time but hybrid learning does. To be more specific, hybrid learning reduces seat time in class. As a result, our model will be better suited to hybrid learning. However, we used the term blended learning because literatures describe hybrid learning is a type of blended learning, and a precise definition remains an ongoing conversation. In comparison to blended learning, the term hybrid learning might have been more extensively used in practice than research as there are less highly cited articles on hybrid learning. According to Stefan Hrastinski, it is important that researchers and practitioners carefully describe what blended learning means to them. He also proposes, researchers should carefully consider while using a more precise, descriptive term as a supplement or replacement for blended learning. So we believe we have adequately addressed this issue.

Has the statistical analysis been performed appropriately and rigorously?

4. Analysis performed as individually randomised trial!

As we have less cluster, we did cluster level analysis. (Also cluster was fixed)

References:

1. Dreyhaupt J, Mayer B, Keis O, Öchsner W, Muche R. Cluster-randomized studies in educational research: principles and methodological aspects. GMS journal for medical education. 2017;34(2).

2. Marion K Campbell, Jill Mollison, Nick Steen, Jeremy M Grimshaw, Martin Eccles, Analysis of cluster randomized trials in primary care: a practical approach, Family Practice, Volume 17, Issue 2, April 2000, Pages 192–196, https://doi.org/10.1093/fampra/17.2.192

3. He Y, Lu J, Huang H, He S, Ma N, Sha Z, Sun Y, Li X. The effects of flipped classrooms on undergraduate pharmaceutical marketing learning: A clustered randomized controlled study. PLoS One.2019; 14(4):e0214624.

5. For tables, please provide footnote on method of hypothesis testing-Provided

6. In table 2, provide the post-hoc comparison for all significant variables-Provided

7. Please clarify, what it means by 1, 2 & 3 in post hoc comparison-Corrected

8. In table 3 regression modelling, how did the variable selection perform? Why only intervention 2 included?

Corrected. Keeping BL as reference, we have modified the model. We have done backward selection ( fit a full model and slowly remove terms one at a time, starting with the term with the highest p-value). Though gender had highest p value, we retained gender since we believe gender has potential role in technology enhanced learning. Next highest p value was Age. We removed it from model. All of the students were in same age group (22-24) and were studying final year pharm D. So we could confidently remove age.

Have the authors made all data underlying the findings in their manuscript fully available?

9. Only summary statistics were provided: Excel data sheet is provided.

D. Is the manuscript presented in an intelligible fashion and written in standard English?

10. Needed substantial revision due to typos (e.g. …website contains 5 chapters per hour…[ page 8]), unclear sentences, and grammatical mistake:

Manuscript edited by professional service.

11. Introduction lacks literature review on the research problem, insufficient background information, and needs to be more structured. All except one reference in the introduction were about flipped classroom approach. The second part of both paragraphs in the introduction better fit in the methods section/discussion section. Objectives can be listed at the end.

Literature review of this topic is published by same authors and provided reference in introduction.

A recent article by Stefan Hrastinski, on “What Do We Mean by Blended Learning?” mentioned that “it is likely that papers that use more specific terms, such as blended synchronous learning or flipped classroom, can be found among the huge number of papers that fits under the inclusive blended learning umbrella”. So may be because of this most of our references from Introduction belongs to Flipped learning”. Also all these cited articles are mentioned about blended learning in introduction or discussion.

We would like to retain second part of both paragraph in introduction. Because that describes why we selected clinical research module and importance of SRL strategies. Discussion modified according to your suggestion.

12. In the methods section, better move the details of web implementation into under “study procedure” section.

Corrected:

13. Discussion needs to be more structured and include relevant references.

Corrected

Reviewer #4: Thank you for the opportunity to review this paper. Generally, the topic would be of interest to many readers. Below are suggestions to address

1. Introduction - Why Pintrich’ motivated strategies for learning questionnaire (MSLQ) is selected?

Motivated strategies for learning questionnaire is most fitting instrument to measure self- regulated learning strategies as it measures cognitive, metacognitive, motivational, emotional aspects of learning procedures. Since this instrument incurred to examine motivational orientation and learning strategy use for students in higher education irrespective of their discipline, we have used this questionnaire to measure self-regulated learning strategies in present study.

Previous research has been conducted to identify most suitable instrument (Systematic search on pubmed and ERIC using keywords self-regulated learning, reflection, questionnaire, instrument and medical or higher education) and found MSLQ is most appropriate instrument and developed for students in tertiary education, regardless of discipline to examine their motivation for learning and their learning strategies

Refference: Soemantri D, Mccoll G, Dodds A. Measuring medical students’ reflection on their learning: modification and validation of the motivated strategies for learning questionnaire (MSLQ). BMC medical education. 2018; 18(1): 274.

Also this questionnaire is free and publically available.

2. Outcome Assessment: How did the items in pre-test and post-test generated? Evaluation of students’ experience and satisfaction – is it newly constructed items or adopted from previous study? Validation?

Questionnaire were developed for the study purpose based on the clinical research module. Questionnaire were validated by 5 experts in the field, for content validity (Questionnaires were validated by 5 experts for its appropriateness, relevance and clarity)

3. Comparison of knowledge score –

-Line 213 - “As shown in table3, mean difference between post-test and pre-test, case studies score, and total score in BL was higher than DT and WEL.”- is it Table 2 or Table 3?- Corrected

- Case study scores – pre or post?- post test, corrected

- Line 224-225 “Students’ pretest score, and whether they were in BL had a significant influence on final score.”- please clarity

Corrected. Keeping BL as reference, we have modified the model. We have done backward selection ( fit a full model and slowly remove terms one at a time, starting with the term with the highest p-value). Though gender had highest p value, we retained gender since we believe gender has potential role in technology enhanced learning. Next highest p value was Age. We removed it from model. All of the students were in same age group (22-24) and were studying final year pharm D. So we could confidently remove age.

4. The paper also includes inconsistencies e.g. decimal points, capital letter/small letter and would benefit from linguistic editing by a native English speaker prior to resubmission-corrected

Reviewer #5: 1. The objective was to evaluate the effectiveness of blended learning on knowledge score using clinical research modules. Assessing knowledge on a specific subject to demonstrate increased competency in that particular field. However, in this study it was assessed only at the basic level, which are insufficient to argue for the value of learning. I would suggest to evaluate the behavior of the students using objective assessment, including direct observation, an interview or a skills demonstration.

A systematic review was conducted on the same topic (Pharmacy education) by same authors and we realized that there are limited number of prospective studies. Also most of the studies conducted from developed nations. Also there are very less number of studies to compare SRL usage between learning strategies. As far as we know, this would be first attempt to evaluate the effectiveness of BL in clinical research modules in pharmacy education

We have conducted focus group discussions to explore the experience of students and included in Pilot study.

We value your suggestions but study was already completed in 2020, submitted closure report to ethic committee and its impossible to get the same students as all are already passed out.

2. The effectiveness of a blended learning environment can be assessed through analysing the relationship between student characteristics/background, design features and learning outcomes. It is always better to determine the significant predictors of effectiveness using learning outcomes as dependent variable.

We value your suggestions but study was already completed in 2020, and its impossible to get the same students as all are already passed out.

3. The language is ambiguous and didn't reflect the context clearly.- corrected.

Attachment

Submitted filename: REviewers comment.docx

Decision Letter 1

Ritesh G Menezes

26 Jul 2021

PONE-D-21-09098R1

Effectiveness of blended learning in pharmacy education: An experimental study using clinical research modules

PLOS ONE

Dear Dr. Thunga,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by 13-August-2021. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A 'Response to Reviewers' letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Prof. Ritesh G. Menezes, M.B.B.S., M.D., Diplomate N.B.

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Academic Editor Comments:

- Address the comments made by Reviewer #2. Refer to the following website (5th criterion for publication): https://journals.plos.org/plosone/s/criteria-for-publication

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

Reviewer #4: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: (No Response)

Reviewer #3: Yes

Reviewer #4: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript can be accepted for publication. The authors have adequately addressed my comments provided during an earlier round of review.

Reviewer #2: Check for grammatical and typographical errors and reform sentence with clear elucidation. The Caption of the tables should be revised and value must be written down to the same decimal place throughout the table.

Reviewer #3: (No Response)

Reviewer #4: Thank you for the opportunity to review this paper again. The authors have addressed comments provided.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Royes Joseph

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachment

Submitted filename: PONE-D-21-09098_reviewer.docx

PLoS One. 2021 Sep 1;16(9):e0256814. doi: 10.1371/journal.pone.0256814.r005

Author response to Decision Letter 1


13 Aug 2021

Dear sir/madam,

Grammatical errors corrected. Sentences reframed with clear elucidation. Table title modified. Decimal places corrected.

References cross checked. No articles retracted.

(Manuscript edited by professional service). Please revert us if you need any additional information.

Thanks & Regards

Girish Thunga

Attachment

Submitted filename: REviewer Comments.docx

Decision Letter 2

Ritesh G Menezes

17 Aug 2021

Effectiveness of blended learning in pharmacy education: An experimental study using clinical research modules

PONE-D-21-09098R2

Dear Dr. Thunga,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Prof. Ritesh G. Menezes, M.B.B.S., M.D., Diplomate N.B.

Academic Editor

PLOS ONE

Acceptance letter

Ritesh G Menezes

23 Aug 2021

PONE-D-21-09098R2

Effectiveness of blended learning in pharmacy education: An experimental study using clinical research modules

Dear Dr. Thunga:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Prof. Dr. Ritesh G. Menezes

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Appendix. Motivated strategies for learning questionnaire (MSLQ).

    (DOCX)

    S2 Appendix. Pilot study details.

    (DOCX)

    S3 Appendix. Various strategies used for different teaching learning method.

    (DOCX)

    S4 Appendix. Web-based e-learning program.

    (DOCX)

    S5 Appendix. Questionnaires.

    (DOCX)

    S1 File. Students demographics, knowledge score & MSLQ score.

    (XLSX)

    S2 File. Feedback file.

    (XLSX)

    Attachment

    Submitted filename: Rebuttal letter.docx

    Attachment

    Submitted filename: PONE-D-21-09098_reviewer.docx

    Attachment

    Submitted filename: REviewers comment.docx

    Attachment

    Submitted filename: PONE-D-21-09098_reviewer.docx

    Attachment

    Submitted filename: REviewer Comments.docx

    Data Availability Statement

    All relevant data are within the manuscript and its Supporting Information files.


    Articles from PLoS ONE are provided here courtesy of PLOS

    RESOURCES