Digital transformation in higher education: A qualitative evaluative study of a hybrid virtual format using a smart classroom system

Abstract

This study presents the salient perceptions of students and instructors in a master's program taught in a hybrid virtual format using the pocket Bipolar Laddering tool, a written open-ended electronic data collection system. Perceptions about the hybrid virtual format were tested on the participants of a master's program taught in the 2021–2022 academic year through a hybrid virtual format based on a Smart Classroom system developed as part of the digital innovations implemented to overcome the COVID-19 pandemic restrictions. This work aims to shed light on the users' salient perceptions of the format, detect the positive elements mentioned by the surveyed participants and identify the negative items in a bid to minimize, or even revert, their effects for future editions of the master. As expected, the findings suggest that one of the main advantages of this format is that it allows students who have difficulty attending classes on campus to enroll on courses. However, the participants detected diverse elements that could be improved such as interaction, the degree of socialization, or the technical problems that arose during teaching sessions. It is hoped that these findings will be of use when adjusting new editions of the program and will help to determine the design and implementation of other hybrid virtual programs in the Institution.

1. Introduction

The Coronavirus Disease 2019 (COVID-19), which became a pandemic in March 2020 [1], forced the world over to adopt a huge number of decisions to cope with the unprecedent circumstances. The initial blanket closure of schools and higher education centers [2,3], was accompanied by diverse educational measures in different countries taken to ensure the continuation of teaching and learning activities [[4], [5], [6]]. To preserve and guarantee the academic activity, diverse institutions promoted a switch from physical face-to-face classes on the campus facilities, to online teaching and learning activities [[7], [8], [9], [10], [11], [12]], a format leveraged by technology, that has been growing worldwide [13]. Technology has played a key role in facilitating alternative digital platforms in all sectors [14], and Information and Communication Technologies (ICTs) have enabled the implementation of remote learning solutions in higher education [[15], [16], [17], [18], [19], [20], [21]]. Therefore, ICTs made the design of innovative teaching and learning solutions to cope with emergency situations feasible [9,11,22,23]. However, the use of ICTs also present diverse threats and challenges [13], such as the use of private data [24] or cybersecurity issues [25]. Specifically, when referring to education, as shown in diverse research works, certain aspects of learning may become an issue, such as, wellbeing [26,27], motivation and interaction levels [19,28,29], readiness for digital learning [30], connectivity issues and hardware and software problems [22,[31], [32], [33]], or privacy and security issues [34].

Participants' opinions of a masters' program delivered through a hybrid virtual format during the 2021–2022 academic course are presented in this research work. The hybrid virtual format meant that the enrolled students could attend the sessions, either physically or remotely in real-time, thus: (1) they were not affected by any further restrictions concerning access to campus facilities; (2) students who initially could not physically attend campus due to diverse circumstances (e.g., living abroad, living far or having problems to travel regularly to the campus facilities, …) could enroll and follow the program. The evaluation of the hybrid virtual format presented in this research aims to analyze the data collected on the participants’ salient opinions: (1) to obtain information about the Hybrid Virtual Format to be added and enrich the SWOT -strengths, weaknesses, opportunities, and threats- analysis [35] of the Institution, to eventually decide whether future academic programs will be offered in the same format; and (2) to redesign the master program.

1.1. Digital transformation in higher education

Higher education is evolving thanks to the digital transformation [[36], [37], [38]] and the strategical approach of educational institutions has changed in many ways [39,40], not just the mere digitization of contents [41]. In fact, the digital transformation is defined according to Vial (2019) as “a process that aims to improve an entity by triggering significant changes to its properties through combinations of information, computing, communication, and connectivity technologies” [42], a definition that clearly refers to a strategical approach. Therefore, according to the definition, it may lead to disrupting traditional business models, which could drive to the implementation or redesign of the specific organization structures [43].

It goes without saying that technology plays a key role in this picture, but just as an instrumental tool, since technology is not an end in itself when talking about the digital transformation in higher education, as shown in diverse research studies [37]. In the end, digital technology enables the transformation of organizations, or more to the point, technology is an instrument to serve the strategy of the educational institutions in terms of mission, vision and objectives [44].

1.1.1. Terminological precisions on learning modalities and activities

E-learning, online learning and distance learning are not the same [[45], [46], [47]]. In this paper, the concept ‘learning modalities’ is understood as “location and timing of interactions” [47]. A simple first distinction that can be made is to classify the modality depending on the location of the student: (1) on campus; or (2) in a remote place of the student's choice. Before the era of real-time data exchange, attending classes in person was the only form of real time interaction between students and instructors, while this kind of interaction was not viable through remote learning. ICT advances and real-time data exchange have paved the way for alternative forms of class interaction, and diverse research studies have focused on taxonomizing the new available options [46,47]; from then on, it was possible, when the suitable technology was implemented, that remote students could interact in real time.

The term Blended Learning (BL) is not equivalent to e-learning [48]. BL is associated with the idea of mixing both, face-to-face (F2F) and online learning activities within the same course experience [17,18,[49], [50], [51], [52], [53]]. Then, BL combines the advantages of the F2F learning with those of the online instruction of students, although BL has diverse specific challenges [50], which are out of the scope in this research often arise.

So, when referring to BL in this article, we refer to mixing face-to-face learning activities and online learning activities -the latter ones carried out through diverse materials (in form of videos, or files) and activities that were uploaded by the instructors in the Learning Management System (LMS)- within the master's program. On the other hand, we use the term hybrid to indicate that the student can choose to attend classes in two ways: (1) physically, in-person, on campus; or (2) remotely, live, through a videoconferencing system.

1.2. The smart classroom system deployed in the campus facilities

A Smart Classroom (SC) system was installed at La Salle-Universitat Ramon Llull (La Salle-URL) in September 2020 [54,55]. Prior to this implementation, in February 2020, an Emergency Remote Teaching (ERT) solution based on the existing LMS had been deployed to cope with the mobility restrictions imposed at the outbreak of the COVID-19 pandemic [11]. In fact, by definition, an ERT solution is thought to be a temporary and provisional solution to cope with unexpected situations [23,56].

To improve the student experience of the ERT system, a SC solution was adopted to try to emulate the face-to-face format as much as possible through a videoconferencing system, which also offered other teaching and learning possibilities. The SC system was based on the deployment of diverse technological equipment in the classrooms and the laboratories, as detailed in Ref. [11]: (1) a smart board, in fact, a computer with a huge touch-control screen; (2) a sound system, including microphones and speakers; (3) an image system, that included two robotized cameras and two TVs; (4) a personal computer (PC) to be used by the instructor, connected to the smart board and to the Internet; (5) Wi-Fi, available for all the class participants that were attending classes in the campus facilities; and (6) the specific software and licenses to allow broadcasting the class sessions through a videoconferencing system. The photographs of Fig. 1, Fig. 2 show the appearance of a typical ‘smart classroom’ at La Salle-URL.

Fig. 1.

Front view of a Smart Classroom at La Salle-URL Campus: smart board, sound system, camera, and TV.

Fig. 2.

Back view of a Smart Classroom at La Salle-URL: sound system, robotized camera, and TV.

Once the SC solution was fully implemented at La Salle-URL Campus, students could attend classes in-person when they were on campus or in real time through the videoconference systems based on the SC technology; and all of them had also access to the recordings of the class sessions. Fig. 3 shows a typical class session imparted in a hybrid virtual format: the instructor and diverse students are physically in the campus facilities, while other students attend the session in person and others remotely. The latter ones are displayed in a mosaic format in one of the TV sets, while the other TV set displays the student that orally interacts with the rest of the participants.

Fig. 3.

Hybrid virtual format: An instructor imparting a class session.

From diverse previous studies [11,33,57], we had information on student perceptions on their experience of the SC system, in relation to diverse academic activities carried out on different academic programs.

1.3. Master in User Experience: designing a resilient format to emergencies

In this section the master program is presented succinctly. Students enrolled on the 60 ECTS (European Credit Transfer System) Master in User Experience (MUX) imparted at La Salle-URL had the option of following a specific format that took advantage of the SC deployment: by either physically attending classes on campus; or by attending online classes in real-time. In this case, the master program had been designed, in terms of teaching strategies, as a hybrid virtual course, a format which enabled students to follow the class sessions through the following diverse formats: (1) live, with the students in the classroom on campus facilities; (2) live, with the students at remote location, which implied attending classes synchronously, through a videoconferencing system; (3) reviewing -or even watching by first time-the content of class sessions through each one of the recorded videoconference sessions whenever and wherever it suited them via a live streaming service. The first two options were synchronous, and thanks to the SC system deployed in the classroom, all students that attended the class sessions live, on campus or remotely, could interact with their peers and with the instructors. In contrast, students who opted for the third format were able to watch the class sessions to review the contents, without any form of interaction. It should be noted that the students had to choose the first or the second option during their enrolment on the program; however, all students could request to attend classes remotely (e.g., when ill, in quarantine due to the COVID-19 …).

Furthermore, the course was structured by combining face-to-face (on campus or remote) teaching and diverse e-learning activities, which constitutes a blended learning approach [53,58]. Both, the e-learning activities, and the recorded class sessions, were available through the LMS of the Institution to all the students. To summarize, the key features that characterize the Master in User Experience are displayed next in Table 1.

Table 1.

Master in User Experience: Key features of the hybrid virtual course.

Class sessions: options	On Campus/Live	Remote/Live	Recorded class sessions
How	Physically in the classroom	Online, through a videoconference system	Streaming recorded class sessions
	Face-to-face, being at the campus facilities	Face-to-face, being remote	Watching recorded class sessions
	Synchronous	Synchronous	Asynchronous
When	According to the class timetable	According to the class timetable	At their convenience, once the recording of class session was available
Interaction	Real time interaction with instructors & with peers	Real time interaction with instructors & with peers	NO Real time interaction
F2F & learning activities	Blended Learning	Blended Learning	Watching recorded class sessions; exclusively e-Learning
	Learning Management System (LMS)	Learning Management System (LMS)	Learning Management System (LMS)

The hybrid virtual format was initially designed to take advantage of: (a) a resilient model, designed as a solution to the COVID-19 mobility restrictions [11,59]; (b) flexibility, since students could choose ‘from where’ and ‘when’ to see and to follow the class contents on their convenience [11,60] in case of not being able to physically attend campus for a justifiable cause, besides being able to review the recorded class sessions whenever they wanted; (c) a student centered approach, since students had access to the LMS content whenever it suited them, an option that may foment personalized learning [61,62].

On the other hand, the adopted format of the course implied diverse challenges: (1) designing specific contents and teaching practices to be performed live, that could be experienced in person by the students, both on campus or remotely; (2) potential interaction issues between instructors and remote students, and also between the on-campus and the remote students [11,63]; (3) adopting an assessment system that was compatible for both on campus and remote students; (4) issues derived from potential technical limitations, such as the internet infrastructure [63].

1.4. Goals of this research

The aim of this research is to gather information about the perceptions of students and instructors on completion of a pilot master program that was imparted in a hybrid virtual format. To perform this evaluative study on this format, a User Experience approach was adopted once the students had completed the course. To do that, the pocket Bipolar Laddering tool was selected, since it facilitates the spontaneous opinions of the users, minimizing possible biases derived from the questions (as explained in section 2.2). While this choice provided us with insights into the perceptions of students and instructors in an open way, we also wanted to know their opinion on aspects that had appeared relevant in our previous research work on videoconferencing [33]. Therefore, four open-ended questions were added to the questionnaire.

From a practical point of view, the opinions collected in this research work were designed to achieve a two-fold objective: (1) to collect the salient perceptions of the participants -students and instructors-on the hybrid virtual format, in order to be analyzed and well-thought-out by the Institution to provide feedback on the format -potentially useful to help decide including (or not), eventual new academic programs in this format-; (2) to obtain feedback for the program coordinators to adjust the master's program, in order to leverage the diverse positive elements detected through the survey, while solving -or minimizing-the negative effects of the issues identified by the participants.

2. Materials and methods

This research methods used a questionnaire answered by students and instructors who had experienced a SC system during their master's program. The students could choose to attend sessions in real time in two ways: face-to-face on the campus facilities or remotely.

2.1. Participants and procedure

The participants were students from La Salle-URL who were enrolled in the User Experience Master's Program in the 2021–2022 academic course. When the students formalized their registration in the program, they had to specify whether they were going to attend classes physically in the campus facilities or connect remotely. However, students that were supposed to attend classes physically also had the option of following the class sessions remotely, provided they presented a certificate that justified their incapacity to physically attend a specific class session. Initially, during the enrollment process, fourteen students (seven women and seven men) opted to attend class sessions physically in the campus facilities, while seven students opted to attend online classes (three women and four men). All the enrolled students in the program were invited via e-mail to complete an open-ended questionnaire on completion of the program. In total, three e-mails were sent to each one of the master's participants, the first one month after the course had ended and the second and the third as reminders, a fortnight and a month later respectively. Responses were voluntary and anonymous, and respondents answered through a Microsoft Forms questionnaire. Since we did not know who had answered the questionnaire, two additional lines were included in the second and third e-mail: (1) thanking the recipient of the e-mail if s/he had already completed the questionnaire; and (2) encouraging the receiver to answer the questionnaire. A user experience tool, the pocket Bipolar Laddering (BLA) was selected to collect the opinions on the hybrid format, as detailed in Section 2.2.1.

The survey was properly answered by 7 students (33.33% of the enrolled students). Some socio-demographic characteristics of the students who responded the survey, are displayed in Table 2.

Table 2.

Sociodemographic characteristics of the students that answered the survey.

Characteristics	Category	Frequency: Occurrence & %
Gender	Female	3 (42.9%)
	Male	4 (57.1%)
Age range	23 to 25	1 (14.3%)
	26 to 30	2 (28.6%)
	31, or older	4 (57.1%)
Course format (enrollment)	On Campus/Live a	5 (71.4%)
	Remote/Live	2 (28.6%)

^aTwo of the surveyed participants attended all the class sessions on campus, while the other three students attending most of the sessions on campus, and a few sessions remotely.

The questionnaire was also answered by diverse instructors that taught class sessions in the master's program. The total number of instructors that were asked to answer was 5, and 4 instructors (80%) responded. It should be noted that the instructors that taught shorter sessions in the program, were not asked to answer the questionnaire. Table 3 shows the information about the instructors that completed the survey.

Table 3.

Information about the instructors that answered the survey.

Characteristics	Category	Frequency: Occurrence & %
Gender	Female	2 (50.0%)
	Male	2 (50.0%)
Number of hours of class in the master program	12, or less	2 (50.0%)
	More than 12	2 (50.0%)

The collected opinions constituted a purposive sample [64,65]. All the participants of the course (except the instructors that taught shorter sessions in the program), were invited to give their opinion on the course. Once all the answers had been received, the researchers checked that they had a maximum variation sample [65]: (1) on campus and remote students (see Table 2), with participants that had followed the sessions exclusively on campus (two students), only remotely (two students), and most sessions on campus and few session remotely (three students); (2) instructors (see Table 3).

The research was designed according to the guidelines accepted when doing a qualitative research work [[66], [67], [68], [69], [70], [71]]. The different steps followed in the research work, consistently with Bengtsson [66], were as follows: (1) To identify the potential respondents, in this case, all the enrolled students in the Master in User Experience and the instructors; (2) To decide the data collection method; (3) To select the analysis methodology to be performed. The first stage was to analyze the data, and three researchers examined the answers of the students to propose an initial codification of the responses. The codification process was performed through an inductive coding approach, which means that the process started with no predetermined codes and generated different codes from scratch based on the collected data set [72]. In a second stage, the three researchers agreed altogether a consensus on the final codification of each answer. This second round constitutes an investigator triangulation, a process which leads to a unique final code for each answer. Triangulation constitutes a procedure that increases the trustworthiness of the findings [66,73,74], since it affects the credibility criterion [75,76]. In the same way, another criterion that also enhances trustworthiness is transferability [75,76], which consists in adding context to the experience that is object of research (see section 1.2).

Informed consent was obtained from all participants involved in the study. Data were collected by a nameless form to be completed on a voluntary basis and treated anonymously. The research presented, as well as the design, collection, and management of its data, was positively evaluated and approved by the Ethics Committee of Ramon Llull University, with the file number CER URL_2020_2021_009.

2.2. Collecting students' and instructors' opinions on a Master's program delivered in a hybrid virtual format through a BLA tool and four open-ended questions

A User Experience (UX) approach provides feedback from users once they have experienced a product or a service [77], which enables researchers to collect and assess their perceptions from the user viewpoint [78], that may differ from the point of view of the designer of the product or service. Several research studies to assess and adjust practices carried out in diverse educational settings have taken advantage of the UX approach, collecting opinions from students or instructors through specific UX tools [11,57,[79], [80], [81]]. The BLA tool can be used to collect user perceptions once they have experienced a product or a service, and in this case it has been used to gather student's feedback about different educational experiences [11,33,57,82].

It should be noted that in this research work a user experience tool has been used to collect feedback from a Master in User Experience. Selecting the user experience approach had nothing to deal with the topic of the master. It was just a coincidence that the pilot program that was object of the research had to deal with the approach and the tool selected to carry out the investigation.

2.2.1. The Bipolar Laddering (BLA) tool

The BLA tool has been described in various research works [11,33,57,[83], [84], [85], [86], [87]], and is based on asking finding out users' opinions from open-ended questions [11,33], which minimize preconceptions or biases, since the BLA tool is conceptually based on a tabula rasa approach [87]. Therefore, participants are asked to give their perceptions, highlighting the different positive and negative elements once their experience as users has been completed. In addition, they are asked to score their level of satisfaction with each one of the identified positive and negative elements. The level of satisfaction is assessed according to the feelings of the user and is ranged from 0 (the lowest level) to 10 (the highest level). The users are also asked to propose their own ideas to improve their experience, which provides valuable information for designers, who can use these suggestions to redesign and improve future experiences [86,87]. Once the researchers collect the data, the positive and negative perceptions are coded through an inductive approach, and each one of the resulting items are listed according to the number of times that each single element has been cited: if an element is cited by more than one time by the participants, this element is labeled as a ‘Common Element’, and the ‘Mention Index’ captures the number of times that the element was cited by different respondents; otherwise, if an element has just been cited by just one participant, it becomes a ‘Particular Element’. Both common and particular elements may lead to future improvements. However, researchers should pay careful attention to the common elements since these items have been perceived in the same way (positively or negatively) by diverse users [86,87]. Fig. 4 displays in a synthetic way the key features associated to the BLA tool. The template that shows the questions included in the pocket BLA (see Figure A1) is also provided.

Fig. 4.

Bipolar Laddering (BLA): key features of the tool.

This tool -Bipolar Laddering-can be implemented through two different modalities: (1) a face-to-face interview, known as a BLA tool; or (2) a written open-ended electronic data collection system, known as the pocket BLA tool [85]. The BLA tool option requires a skilled interviewer an also more time than the pocket BLA option but allows asking further questions by the interviewer to clarify the comments that the user is explaining spontaneously. The pocket BLA version consists in presenting an open-ended questionnaire to be autocompleted by the user; by design, it does not offer the possibility to clarify any doubts that could appear in the collected answers since there is no interviewer, but this method does not make it possible to survey many users at the same time, as carried out in diverse research works [11,33]. Furthermore, one of the advantages of the latter possibility is that it also facilitates the digital responses to the survey, which access to the collected data immediate. Therefore, depending on the approach and the goals to be researched, the researcher can choose to select the best option in each case: a BLA or a pocket BLA.

The BLA tool enables us to collect spontaneous information from the surveyed user, avoiding potential biases when questioning about a particular topic. The question that the students were requested to answer was: “Once you have finished your studies on the Master in User Experience at La Salle Campus Barcelona, list three elements that you consider positive associated with the use of the Smart Classroom system that have made it possible to have students On Campus (in person, F2F in the classroom) and off campus (remote), both synchronously”. A second question was raised about the negative elements of the experience. To enrich the perceptions about the class format, a similar process asking a similar question, using the pocket BLA tool once more, was addressed to the instructors that imparted class sessions in the master program: “Once you have finished the Master in User Experience at La Salle Campus Barcelona (academic year 2021–2022), list 3 positive elements that you would associate with the use of the Smart Classroom system that have allowed students to be On Campus (in person in the classroom) and off campus (remote), both synchronously”. Similarly, a second question was asked about the negative elements on the teaching experience.

2.2.2. Four specific open-ended questions about the course

Once the surveyed participants had answered these questions, they were specifically asked four additional open-ended questions in case they had not answered the pocket BLA tool. The questions were: (Q1) What is your opinion on how the collaborative work has been undertaken by the students of the course using the Smart Classroom system?; (Q2) What are the advantages of attending class remotely presents compared to doing it in person?; (Q3) What are the disadvantages of attending class remotely compared to attending in person?; and (Q4) What could be done to improve the experience of students attending remotely?

3. Findings

Diverse requests were sent via e-mail to collect the students' and instructors’ opinions through two online questionnaires, designed to allow us to cluster the answers in two groups: students and instructors. In this section the answers are presented through the format that is associated to a BLA tool for each of the groups. The analysis of the collected answers is presented in Section 4.

Once the researchers analyzed and agreed on the categorizations of all the answers given by the students, the positive elements were synthesized in Table 4. It should be noted that several participants included diverse elements when asked to write a positive or a negative item. In Table 4, and under the label sPcE (students' Positive common Elements), all the positive elements named by more than one student are shown. On the other hand, the label sPpE (students' Positive particular Element) identifies diverse the positive elements that were suggested uniquely by a single student. In addition, the table includes columns to reflect: (1) the average score associated by each element -which gives an idea of the user satisfaction in numbers-, and (2) the mention index (‘Mention I.‘), which reflects the total number of surveyed students that cited a specific element. Table B1 displays all the different scores given by the participants.

Table 4.

Students: Positive (P) common (c) and particular (p) Elements.

Item	Description	Average	Mention I.
sPcE_01	Accessing recorded class sessions	8.6	5/7
sPcE_02	Attending classes in real-time	9.0	4/7
sPcE_03	Allowing students to remotely attend online classes	9.0	3/7
sPcE_04	Flexibility to attend the class sessions	10	3/7
sPcE_05	Quality of the videoconferencing system	9.5	2/7
sPcE_06	Enabling students to remotely attend online classes from abroad	8.0	2/7
sPpE_01	Accessing information in the best way as possible	9.0	1/7
sPpE_02	Enabling in-person interactions with students and instructors	8.0	1/7

User Experience: Findings from the pocket BLA tool. Positive elements. Sorted by Mention Index.

In Table 5 all the negative elements identified by the students are presented in the same format and following the same pattern that was applied in Table 4. Therefore, in the table are reflected the sNcE (students' Negative common Elements) and the sNpE (student's Negative particular Elements). Table B2 displays all the different scores given by the participants.

Table 5.

Students: Negative (N) common (c) and particular (p) Elements.

Item	Description	Average	Mention I.
sNcE_01	Issues when framing the instructor through the cameras	3.0	3/7
sNcE_02	Feeling of lack of inclusion when attending classes remotely	3.3	3/7
sNcE_03	Accessing recorded class sessions limited to a short period of time	1.0	2/7
sNcE_04	Technical failures of the system	2.5	2/7
sNpE_01	Not having the option of splitting the screen to have the view of 2 cameras	2.0	1/7
sNpE_02	Timetable (for people that are abroad in South America)	2.0	1/7
sNpE_03	Poor time distribution between the different seminars	4.0	1/7
sNpE_04	Difficulty in using the system	1.0	1/7
sNpE_05	Some instructors did not answer the e-mails	1.0	1/7
sNpE_06	Being online: when ‘raising the hand’, sometimes not being attended	4.0	1/7
sNpE_07	Some discrepancies between the syllabus and what was required in class	4.0	1/7
sNpE_08	Information on deliverables: it was only in ECT (European Central Time)	4.0	1/7

User Experience: Findings from the pocket BLA tool. Negative elements. Sorted by Mention Index.

Regarding the answers to the four open-ended questions when experiencing the class format through the SC system, students’ responses were as follows:

Concerning question Q1, about students' perceptions regarding the interaction between the participants that were carried out through the SC system, five answers highlighted that the interaction worked well (“very good”; “in the end, it worked fairly well”; “the idea worked very well”; “it worked very well because all participants were professionals, despite the platform not being intuitive enough”; “it worked really good; good usage of the tool”). Lastly, it can be mentioned an answer that reflected some skepticism about the question (“we did not interact too much through the SC system”; “I do not have an opinion about the question”).

Question Q2 was about the perceived advantages of attending classes remotely over doing it in person in the campus facilities. The collected answers were: “it was easier to write down notes during the class sessions”; “flexibility”; “not too much; the unique advantage is to save time, since you do not need to spend time since commuting to the campus facilities is not needed”; “it allows enrolling in courses abroad”; and “I was enrolled on the campus option; I do not appreciate any advantage in attending class sessions remotely”. On the other hand, the next question -Q3- was about the disadvantages of the format which are listed, as follows: two opinions stressed that the class sessions were much difficult to follow (“it is exhausting being in front of a screen paying attention continuously”; “it is more difficult to grasp all the details than attending classes in the campus facilities, so classes are not as easy to follow”); “being remotely you lose all the advantages of attending class sessions in the campus facilities”; “an important part of the educational experience is lost”; “you do not have physical contact with the other participants in the course”; and “it is not easy for people abroad in South America to wake up early in the morning to follow live the class sessions”.

Regarding what could be done to improve the experience when attending the class sessions remotely (Q4), the answers were: “really, I do not know; I believe that the instructors did all they could to make it a great experience for the students that attended class sessions remotely. I attended classes in the campus facilities”; “establish more efficient methods to enhanced interaction with the students that attended class sessions remotely; make the use of cameras mandatory -and keep the cameras on-when attending class sessions remotely”; “being more flexible with the timetables, when checking students’ class attendance”; and “previously to the class sessions, uploading the pdf files so students can follow the sessions in an easier way”.

Table 6 presents the instructors’ perceptions of the positive elements, opinions that were collected by means of a pocket BLA tool.

Table 6.

Instructors: Positive (P) common (c) and particular (p) Elements.

Item	Description	Average	Mention I.
iPcE_01	Facilitates teaching in-person and remote as a single group	9.8	4/4
iPcE_02	Nearly no differences teaching students that are in person or remote	9.3	3/4
iPcE_03	Integration of students	9.5	2/4
iPpE_01	Easy to use and intuitive	10	1/4
iPpE_02	Practical	10	1/4
iPpE_03	Effective	10	1/4
iPpE_04	Timetable	10	1/4
iPpE_05	People that live abroad/far can enroll in the course	9	1/4
iPpE_06	Smart board: it gives a lot of options	8	1/4

User Experience: Findings from the pocket BLA tool. Positive elements. Sorted by Mention Index.

In Table 7 all the negative elements identified by the instructors are presented, as follows:

Table 7.

Instructors: Negative (N) common (c) and particular (p) Elements.

Item	Description	Average	Mention I.
iNcE_01	Some online students ‘disconnected’ during a while	3.5	2/4
iNcE_02	Problems with the Internet connexion	2.0	2/4
iNpE_01	Interaction with remote students	3.0	1/4
iNpE_02	Class dynamics	3.0	1/4
iNpE_03	Affinity with face-to-face students	2.0	1/4

User Experience: Findings from the pocket BLA tool. Negative elements. Sorted by Mention Index.

The responses to the four open-ended questions given by the instructions were:

The answers to question Q1, were as follows: “Very satisfying. I did not find any difference between the collaboration between face-to-face and online students”; “A very good system for students and instructors”; “Very good, it gives the instructors a lot of possibilities, since the whole system allows to handle small groups of students”; and “The dynamics have gone very well, but it is important to emphasize that these are master's students; it would not work the same way with undergraduate students”.

Q2, about the perceived advantages of attending class remotely over doing it in person in the campus facilities, obtained these answers: “It gives the students peace of mind when facing the possible loss of a face-to-face classes and, on the other hand, it is enriching that students from other countries, who otherwise would not be able to attend the masters’ degree are able to”; “An advantage for students, by allowing them to overcome physical attendance problems in the classroom”; “Students can save time -they do not need to commute- and can optimize their agendas” and “Students in remote format can better manage their time, without depending on the transfer to the University”. On the other side, the disadvantages (Q3) that were mentioned by the instructions were: “None especially relevant according to my perception”; “Sometimes, interaction with remote students was difficult”; “Difficulties in focusing and keeping concentration levels high when attending classes remote”; and “Problems with the link with instructors and the fact of belonging to a university”.

The question about how the experience when attending class sessions remotely could be improved (Q4) was answered as follows: “Improving the internet connection”; “Think about how to improve interaction with students that attended classes remotely”; “Trying to ensure that when students interact through teamwork, all of them work in a remote format”; and “Train instructors in effective hybrid methodologies”.

4. Discussion

This research is focused on analyzing the participants' perceptions on a hybrid virtual format. Through a pocket BLA tool, students' and instructors’ salient opinions about the format were collected to identify the strengths and weaknesses of the hybrid virtual format, as carried out in other research works focused on education [11,33,57,82]. First, a discussion about the findings derived from the pocket BLA is carried out.

When analyzing the positive elements associated with the Hybrid Virtual Format through videoconferencing based on a SC system deployed in the classroom, students appreciated the most the possibility of watching the sessions of class that had been previously recorded (sPcE_01), in line with other research works [33,88]. This option, in fact allowed students: (1) to review what had been explained in class, in case that they had doubts or misunderstandings about what had been explained, or simply to review the contents that were explained by the instructors during the sessions of class; or (2) to watch the session of class, in case that the student had been unable to could attend the live session of class. One limitation of watching the recorded classes was the following one: the content could be reviewed, but there was no possibility of interacting with instructors in order to clarify and solve potential doubts. In addition, the negative element most mentioned by the students was associated with the recorded class sessions: the fact that recorded sessions were only available for a limited period -a week- (sNcE_03), instead of being available forever.

Two other elements of the analyzed format were highly appreciated: Attending classes live (sPcE_02), which enables the student to interact with the instructors and also with their peers; and also, the possibility of attending classes when being remote (sPcE_03), especially when living abroad (sPcE_06 and iPpE_05). Attending classes synchronously remotely, offers the possibility of following everything that is happening during the session of class in real time. In fact, it is someway similar to physically being in the classroom -students can ask questions, discuss topics … -, which lets students take an active role through in-person interactions with other students and/or instructors (sPpE_02). The other positive element was the possibility of following the lectures remotely, which entails (for the institution) targeting a greater number of potential students, since people that live abroad – or significantly far from de campus facilities-can be enrolled on the program, since they can attend classes in a remote format (iPpE_05). In the analyzed program, several students attended classes remotely from South America, while the campus facilities of the educational institution were in Europe, at La Salle Campus Barcelona. In addition, some students that lived in the same country, but quite far from the campus -at more than 100 km-, opted to enroll on the program attending classes in the remote format.

When referring to the technical elements of the SC system, on one hand, the quality of the system (image, sound) was highly appreciated by the students (sPcE_02), and on the other hand, there were complains of the students about how the instructor was framed by the cameras by the robotized SC system (sNcE_01). Issues related with the internet connection were mentioned by two instructors (iNcE_02), while other technical problems were identified by two students (sNcE_04), and one student complained about the difficulty in using the system (sNpE_04). However, one of the instructors claimed that the system was easy to use and intuitive (iPpE_01), practical (iPpE_02) and effective (iPpE_03). The options derived from the use of the smart board -in fact, a specific feature of the SC system-was also highlighted by one of the instructors (iPpE06). Therefore the trade-off is positive in this block referred to the technical elements of the system, despite some problems about the internet connection, a topic also highlighted in other research works [22,[31], [32], [33]].

One controversial finding is that instructors and students disagree on the how effective online classes are. According to the opinion of all the instructors, on campus and remote students may be taught as a unique group (iPcE_01), despite one of the instructors pointed out some difficulties when interacting with students that attended classes remotely (iNpE_01) while the other three instructors appreciated nearly no differences in teaching students (iNcE_02); in fact, interaction between students was seen as a positive element by one student (sPpE_02), while two instructors highly assessed the integration of the students (iPcE_03). Conversely, diverse students complained about certain deficits that they appreciated when attending classes remote, such as “Feeling of lack of inclusion when attending classes remotely” (sNcE_02) or “Being online: when ‘raising the hand’, sometimes not being noticed” (sNpE_06). In this block of opinions, it should be mentioned that at times, it was noticed by two instructors that students in remote mode found it difficult to maintain a high level of attention (iNcE_01) -one of the instructors noted that “some online students ‘disconnected’ for a while”-, which in fact create difficulties for the instructor.

Another negative comment was about the timetables, as mentioned by sNpE_08 and iPpE_04. As previously mentioned, the campus facilities were in Europe (ECT -European central time-), while some student attended classes from South America in real time. It implied a trade-off to teach classes at a reasonable hour in both countries which also affected the deadlines fixed to deliver the assignments. It should be noted that a single date and timetable was published when referring to a specific task, to avoid confusions. However, some students complained about this decision, as previously mentioned (sNpE_08).

The answers to the open-ended questions focused on four specific topics are presented as follows. Regarding the first question, which was referred to the collaborative work -in fact, rate the interaction between students when carrying out teamwork-, most of the students manifested that interaction worked reasonably well, although one student was a little skeptical about it, while saying that he did not interact too much through the SC system. Instructors gave similar answers positively assessing the interaction that they had with the students with remote an on-campus students. In addition, an instructor that the SC system fomented class dynamics with small groups of students. However, when analyzing the answers given to the fourth question, which was about what could be done to improve the remote experience, students proposed the creation of better mechanisms to enhance the interaction with students that attended classes remotely. Here instructors proposed that they received effective training in hybrid methodologies to perform better class sessions. The second open-ended question was about the advantages that they perceived when attending classes remotely. Students highlighted three items: being able to write down ideas in an easier way; flexibility and saving time by not having to commute; and their preference to face-to-face classes, since they saw no advantage in attending classes remotely. Instructors gave similar answers (flexibility and time-effective), while adding that the remote format enabled students that lived abroad to attend class. On the other hand, the third question was focused specifically on the disadvantages that the students perceived when studying remotely to which the common response was that it was more exhausting than face-to-face classes, and the learning experience was poorer than being physically in the campus facilities. Two instructors stressed that the students that attended classes through the remote format found it difficult to remain concentrated and focused; in addition, one of the instructors added that the bonds between the students and the teaching institution were more difficult to foment when attending classes remotely.

The findings of this research show that the hybrid virtual format worked well according to the students' and instructors’ perceptions. The possibility of broadcasting the sessions of class through videoconferences creates a more resilient teaching format, since classes can continue to be delivered even in case of emergencies that eventually could restrict -or even forbid-the access to the campus facilities. Remote teaching was appreciated by students -since they could select one or the other possibility, despite some research works found that students preferred the face-to-face format in the campus facilities in front of the online format [11]. The hybrid virtual format theoretically joins the advantages of teaching and learning face-to-face in the campus facilities and remote, since both modalities are carried out synchronously and at the same time. However, when the two modalities are performed at the same time, the instructor must adapt the whole course to both modalities, and even though theoretically are quite similar, they are not exactly the same. Therefore, the instructor must take great care to design activities to minimize the effects that can jeopardize the educational experience due to the issues identified in this research (internet limitations, eventual technical problems, distractors, interactions, sense of belonging, …), proposing solutions to combat the videoconference fatigue [[89], [90], [91], [92]], or even establishing mechanisms to incentive that the students maintain their cameras connected when being remote [93,94].

Deploying a SC system at La Salle-URL was a better solution in terms of performance than the initial Emergency Remote System that was deployed at the very beginning of the outbreak of COVID-19 as an answer to cope with the restrictions imposed [11]. Once the SC system was implemented in the classrooms and in the laboratories of the campus facilities, a strategical approach was implemented to take advantage of all the available technology. In fact, all the technology that has been deployed to cope with the limitations imposed because of the COVID-19 pandemic -an improved LMS and the SC system-constitutes a strength -in terms of SWOT analysis-of the Institution. An analysis of the perceptions of both -the students and the instructors that have experienced the hybrid virtual format-may help consider the opportunity - in terms of SWOT analysis and of business portfolio-of offering educational programs in the hybrid virtual format. Nevertheless, findings from this research should be considered when analyzing the strategy of the organization, in order to take advantage of the technological advances that have emerged from the digital transformation.

Findings show that the hybrid virtual format is a feasible option to be included in the business portfolio, as it mixes all the campus and remote students at the same time. In case of an emergency that could eventually impede attending classes physically in the classroom, this format could be a valuable solution. However, this format constitutes a challenge for both students and instructors. Diverse issues arise when interacting participants are remote -e.g., interaction between the participants, timetable issues … -; in fact, interaction is mentioned as an issue when attending online classes in other research [95]. Offering specific training to the instructors in order to provide them the proper techniques to teach students that are on campus and those who connect remotely at the same time, could be very useful to cope with one of the negative elements identified by the students in this research work. In addition, a careful design of the first session of class could help establish a dynamics oriented to interaction between all the participants -students and instructors-, since the first day of class has an impact on the climate and the dynamics of the course, including the students’ engagement, as found in diverse research works [[96], [97], [98], [99]]. Implementing the hybrid virtual format would open the possibility to offer programs to people that live abroad, which would consequently increase the number of potential students that could enroll on the programs. Since the content to be taught in both kinds of programs could be the same, the adoption of solutions that could take advantage of the LMS -such as flipped classrooms, or audiovisual material to be seen after the class sessions- and the SC systems -such as recorded class sessions … -, which could provide: (1) a resilient format which constitutes an insurance to keep on providing the class sessions in case of global emergencies, (2) flexibility for the students since they should have diverse ways of following the class sessions, and (3) a certain degree of personalization in terms of content thanks to the content included in the LMS. Consequently, this hybrid virtual format would result in a solution that would increase the value proposition offered to the student. Therefore, this hybrid virtual format would imply that a higher the number of students could be enrolled on the programs, which would result in increasing the total revenues.

The limitations identified in this research study are presented, as follows. The first derives from the fact that the survey questions were deliberately sent out one month after the final session of the course so that all the students had fully completed their experience -as required in a user experience approach- and had received their final grades. This lapse of time -from finishing the program to receiving the survey-may have had a negative impact in the number of received answers, since some students may have ‘switched off’ from their learning experience. Performing the survey during the last session of the program could have obtained more responses. Though, as said, the whole experience would have not been evaluated in this case. In this case the 42.3% of the participants -which includes students and instructors-have given their opinion about their experience. However, the collected opinions constitute a purposive sample and the number answers is not an issue [100], since the salient items of the experience can be identified. And the goal of this research work was to obtain the salient perceptions once the whole experience had finished, in order to carry out a complete analysis about hybrid virtual formats. The second limitation that we have identified is that the collected opinions come from a specific discipline. Further research should be done implementing the same hybrid virtual format to check if the perceptions of the participants are similar.

5. Conclusions

The aim of this research was to discover the salient perceptions of both, the students and instructors of a master's program imparted in a hybrid virtual format, in order to obtain insightful information for the Institution. The findings of this research also are useful to help improve the future editions of this specific master program. Students highly appreciate this format, since it allows them to follow a master's program live and synchronously from abroad, a feasible option enabled by the hybrid virtual program. However, diverse issues were identified as elements to improve, such as the interaction between the participants -affecting all the students-, the need to be connected at untimely hours to follow the session of classes -remote students- or dealing with technical problems -remote students-. Instructors did not find interaction issues and considered that the format constituted a great experience. However, they perceived issues related with the internet connection, and that some remote students were not as engaged as the students that were physically attending the sessions in the campus facilities. The format, by design, allows teaching synchronously to students that opted to attend classes remotely, through a videoconferencing system. Hence, thanks to the digital transformation crystalized in a SC deployment has resulted in: (1) a resilient format that enables the Institution to continue teaching and learning, which is robust enough to cope with eventual access restrictions to the campus facilities; and (2) a live format that could potentially increase student intake on courses, since the physical presence in the campus facilities is no longer a requirement. Therefore, a strategical approach based on the findings obtained from this research could be conducted by the institution, in order to consider to what extent this hybrid virtual format can be applied to other programs, and also how it can be improved on this particular master's program.

Production notes

Author contribution statement

Josep Petchamé: Ignasi Iriondo: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Oihane Korres: Conceived and designed the experiments; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Jessica Paños-Castro: Performed the experiments; Analyzed and interpreted the data; Wrote the paper.

Funding statement

This research was funded by the project “Challenges in University Teaching in the Post-COVID’19 Era (CUT-POSTCOVID)”, granted at the VIII Call of ACM (Aristos Campus Mundus) Research Projects 2022, with the grant number ACM2022_09.

Data availability statement

Data will be made available on request.

Additional information

No additional information is available for this paper.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A.

Fig. A1.

Pocket BLA: Template in an excel format to be fulfilled by the participants about their experience once completed the Master in User Experience.

Appendix B.

Table B1.

Participant responses: Positive elements. Score associated to each item.

Item	S1	S2	S3	S4	S5	S6	S7	I1	I2	I3	I4
sPcE_01		8	9	9	8		9
sPcE_02				9	10	8	9
sPcE_03		10	7	10
sPcE_04		10	10				10
sPcE_05	9				10
sPcE_06		10		10
sPpE_01	8
sPpE_02						8
iPcE_01								10	9	10	10
iPcE_02								10	9		9
iPcE_03									9	10
iPpE_01								10
iPpE_02								10
iPpE_03									9
iPpE_04										8
iPpE_05										9

Participants (first row of the table): S# stands for student. I# stands for instructor.

Table B2.

Participant responses: Negative elements. Score associated to each item.

Item	S1	S2	S3	S4	S5	S6	S7	I1	I2	I3	I4
sNcE_01	1	3		5
sNcE_02		5	3				2
sNcE_03				1		1
sNcE_04			3				2
sNpE_01	2
sNpE_02		2
sNpE_03						4
sNpE_04					1
sNpE_05						1
sNpE_06				4
sNpE_07					4
sNpE_08					4
iNcE_01									4		3
iNcE_02								2			2
iNpE_01											3
iNpE_02									3
iNpE_03									2

Participants (first row of the table): S# stands for student. I# stands for instructor.