Emergency remote teaching and students’ academic performance in higher education during the COVID-19 pandemic: A case study

Abstract

The COVID-19 pandemic has caused a massive disruption in the way traditional higher education institutions deliver their courses. Unlike transitions from face-to-face teaching to blended, online or flipped classroom in the past, changes in emergency remote teaching –a temporary shift of instructional delivery to an alternate remote delivery mode due to crisis circumstances– happen suddenly and in an unplanned way. This study analyzes the move to emergency remote teaching at the School of Telecommunication Engineering (Universidad Politécnica de Madrid), and the impact of organizational aspects related to unplanned change, instruction-related variables –class size, synchronous/asynchronous delivery– and use of digital supporting technologies, on students' academic performance. Using quantitative data of academic records across all (N = 43) courses of a bachelor's degree programme in Telecommunication Engineering and qualitative data from a questionnaire delivered to all (N = 43) course coordinators, the research also compares the academic results of students during the COVID-19 pandemic with those of previous years. The results of this case study show an increase in students' academic performance in emergency remote teaching, and support the idea that organizational factors may contribute to successful implementation of emergency remote teaching; the analysis does not find differences across courses with different class sizes or delivery modes. The study further explores possible explanations for the results of the analysis, considering organizational, individual and instruction-related aspects.

1. Introduction

In 2020, the impact of COVID-19 has been noted in practically all areas of activity, but its effect has been particularly strong in teaching and learning. The pandemic has shaken up the landscape of higher education worldwide, with responses to the pandemic from higher education institutions generally falling into three categories (Hodges, Moore, Lockee, Trust, & Bond, 2020; Smalley, 2020): maintaining in-class teaching with social distancing, creating hybrid models (blended learning, limitation of students in campus) or moving to online instruction.

In Spain, where the predominant teaching modality across universities is face-to-face learning –only 15 percent of Bachelor's Degree students are enrolled in public or private distance education universities (EDUCAbase, 2020)–, the declaration of the state of alarm in the nation and the enforcement of total lockdown by national authorities (Real Decreto 463/2020, 2020) in an attempt to control the virus spread forced all face-to-face universities to move to online instruction, which required changing the teaching methods and resources to adapt them for distance education.

Three days before –March 11, 2020, coinciding with the declaration of the COVID-19 as a pandemic (World Health Organization, 2020)–, all face-to-face education activities were suspended in the Autonomous Community of Madrid (Order 338/2020, 2020). This suspension occurred barely four weeks into the second semester. With over 98 percent of undergraduate students attending face-to-face learning in Madrid (EDUCAbase, 2020), the impact of the pandemic was even more dramatic. Mostly unprepared, higher education instructors had to make the necessary changes and adjustments overnight. Of course, this situation has not been exclusive to Spain, as Crawford et al. (2020) show in their review of responses to the COVID-19 situation across 20 different countries, noting that the move to online teaching has occurred only in some cases –mostly European countries (Crawford et al., 2020).

Existing literature on the transition from face-to-face teaching to blended, online or flipped classroom learning study these changes under the premise that the instructional changes are carefully planned by the instructors. Most often, the move to online teaching is carried out voluntarily by the teaching staff with help from support personnel. It is a process that takes both resources –human, intellectual, technical– and time: it is estimated that adapting a typical course to online teaching (including planning, preparation and development) takes between six and nine months (Hodges et al., 2020). However, research on how to make these sudden transitions rapidly, and the potential effects of the decisions taken by institutions and instructors regarding the use of different instructional methods or supporting technologies, means (or at least meant before COVID-19) venturing into uncharted territory.

When sudden transitions as a response to a crisis occur, coordinated measures would take too long to put into place, especially when the rigidity of bureaucracy constrains the ability to change by establishing rigid rules (Haveman, 1992). Therefore, the decision on which adaptation strategy to make is left to individuals; in this case, instructors who have to swiftly select among multiple digital tools with different capabilities to support teaching while balancing their workload. More particularly, some of these decisions include the support of asynchronous –e.g. content management systems, message boards, e-mail, pre-recorded videos of class sessions– or synchronous –e.g. chat, videoconferencing or real-time collaboration systems, instant messaging– tools and may even involve changes in the assessment activities or assessment criteria. This research investigates the impact of such choices in academic performance to identify successful transitioning strategies.

As a word of caution, it has been argued by experts that the teaching modality offered as a result of the transition to digital spaces caused by COVID-19 cannot be labelled as ‘online learning’, and thus a new concept has emerged to define the new situation: ‘emergency remote teaching’ (Hodges et al., 2020; Milman, 2020; Rapanta, Botturi, Goodyear, Guàrdia, & Koole, 2020; UoPeople, 2020). Hodges et al. (2020) indicate that the main difference between online learning and emergency remote teaching lies in that online learning results from careful instructional design and planning, requiring an investment in a whole ecosystem of learner supports that takes time to build, whereas emergency remote teaching emerges as a response to a crisis and entails a temporary shift of instructional delivery to an alternate delivery mode that involves the use of fully remote solutions for instruction that would otherwise be delivered using face-to face, blended or hybrid courses. The key term then is 'temporary', as emergency remote teaching assumes that teaching will return to the original format once the crisis ends.

If we observe the changes in teaching and learning caused by COVID-19 under the lens of emergency remote teaching, it could be argued that most, or at least part, of the findings from existing research on online learning might not be applicable to this situation. However, it is also true that the decisions that instructors have had to make to deliver their courses are not that different from the intervention features (Means, Bakia, & Murphy, 2014) or learning design options (Hodges et al., 2020) they have to choose when they plan, design and implement an online course. Admittedly, some of these options, such as breadth (whole program, course, portion of course, brief episode) or modality (blended, semi-blended) are imposed by the pandemic situation and cannot be really chosen, but instructors do have some degree of freedom in their decision about the remaining ones (e.g. online communication synchrony, student-instructor ratio, role of summative assessments, etc.).

With these nuances under consideration, this research study aims to answer three main research questions:

• RQ1: How have instructors adapted their teaching of graduate courses to emergency remote teaching under the COVID-19 pandemic in a specific context (the School of Telecommunication Engineering at Universidad Politécnica de Madrid)?

• RQ2: Are there any differences in students' academic performance between the courses delivered in emergency remote teaching and traditional face-to-face courses?

• RQ3: Are there any differences in students' academic performance depending on the different instructional decisions made by the instructors in emergency remote teaching?

To answer these research questions, the study describes the experience of the changes undergone by the School of Telecommunication Engineering at Universidad Politécnica de Madrid as a result of the COVID-19 pandemic, explores the potential impact of different instructional decisions (online communication synchrony, number of students, digital technologies used) in the academic results of students enrolled in 43 Bachelor's Degree courses, and confronts these results with those from the two previous academic years.

The remainder of this document is structured as follows: Section 2 reviews the literature on organizational aspects that might affect the outcomes in cases of sudden, unplanned changes, as well as literature on differences in academic performance between online and face-to-face learning, and describes the main supporting technologies for emergency remote teaching. Section 3 frames the concepts developed in the previous section in the context of this research study. Section 4 details the method used in the empirical research, which is followed by the presentation of the data analysis and results in Section 5. Section 6 discusses the main findings from the research and aims to explain the results of the analysis, and Section 7 outlines the main limitations of the study.

2. Literature review

2.1. Unplanned change: Organizational aspects

Unplanned change is the response to a need for action precipitated by unanticipated events or crises (Knowles & Saxberg, 1988). To study unplanned change, we turn to the organizational ecology theory (Hannan & Freeman, 1977), based on the theory of punctuated equilibria (Eldredge & Gould, 1972) in evolutionary biology. The main idea of the ecological model of organizations is that organizations are subject to structural inertia, which limits their ability to rapidly adapt to changes (Haveman, 1992). Under both the biological and organizational theories, sudden transformations are denominated punctuational changes, which operate under the structural reordering of environmental conditions.

From an organizational view, we may then ask: what factors may affect success when an organization faces punctuational changes? First, organizational change may be beneficial if two conditions are met: that the changes occur in response to dramatic environmental shifts (punctuational change), and that they build on established routines and competences (Haveman, 1992). In the case of higher education institutions and the COVID-19 outbreak, the first condition is met, whereas the second depends on the decisions made by the instructors. These decisions depend on the familiarity of instructors with online learning and supporting technologies, as well as the compatibility of the new situation with the instructional methods that they have been using over the years. Haveman (1992) further emphasizes that under sudden environmental changes simply reproducing previous ways of doing things may lead to failure. Additionally, the degree of diversification in how the organization reacts –changes across three dimensions: the clients it serves, the goods and services it produces, and the technologies it employs– affects performance in a way that the more closely the new activities are related to the previous ones, the greater the probability of success.

The structure of the organization also has an impact on the potential success of the change. Literature on organizational change agrees on identifying bureaucracy as one of the primary barriers to the implementation of changes. While bureaucratic systems are effective in implementing planned change (Knowles & Saxberg, 1988), they are generally not efficient in responding to unplanned or punctuational changes because of the numerous rules under which they operate and their natural tendency to resist to innovation (Haveman, 1992; Knowles & Saxberg, 1988). A final important aspect to consider is the ability of an organization to foster the existence of an organic organization, based on informal relations and where communication is diffused among members in a non-centralized way, as such kind of organizations have higher capabilities to cope with changes (Knowles & Saxberg, 1988).

2.2. Academic performance in face-to-face versus online learning

Keeping in mind the difference between emergency remote teaching and online learning described in the introductory section, but also the similar elements of decision when moving teaching to the online space, this subsection explores the differences found in prior research related to academic performance between face-to-face and online learning.

The study of the differences in student achievement –measured as student final grades– between face-to-face, blended, and online learning has been a central topic in educational research for decades. The results of these analyses vary and seem to be extremely dependent on the type of analysis and the sample of the study. For instance, the results from the analysis of single courses may offer interesting but anecdotal evidence of these differences due to many different potential confounding variables –e.g. Urtel's (2008) finding that students perform better in face-to-face instruction. As the number of courses under analysis increases, however, the results seem to confirm that students obtain higher grades in online learning compared to those in face-to-face instruction, even though the difference is negligible –e.g. Ladyshewsky's (2004) analysis of 9 course units or Cavanaugh and Jacquemin's (2015) study on 5000 courses.

An alternative view is offered by meta-analyses. Existing meta-analyses tend to support the idea that either the academic performance of students –as final course grades– in online learning is higher than in face-to-face courses, or there are no significant differences between both. For instance, Shachar and Neumann (2003) found that distance education outperforms face-to-face learning; Zhao, Lei, Yan, Lai, and Tan (2005) found no significant differences in student outcomes but also warned that a large number of factors vary from one study to another; Jahng, Krug, and Zhang (2007) found no significant differences for aggregated undergraduate and graduate courses, but also that graduate courses were significantly less effective in online learning than in face-to-face modes, while the opposite occurred in undergraduate courses; and Means, Toyama, Murphy, and Baki (2013) found that students performed modestly better in online learning conditions when considering both ‘pure online’ and blended learning, but the differences were only sustained when comparing blended and face-to-face learning –in other words, there were no significant differences between ‘pure online’ and face-to-face learning.

More recently, the focus has shifted to the comparison of blended and face-to-face learning, with results supporting higher achievement in blended learning (Bernard, Borokhovski, Schmid, Tamim, & Abrami, 2014; Vo, Zhu, & Diep, 2017) and a moderating effect of different variables, such as the kind of computer support used, interaction treatments or whether the courses belong to STEM or non-STEM disciplines –higher effect is found in STEM disciplines (Vo et al., 2017). From the above, we might expect either no significant differences between courses from previous years and the same courses delivered in emergency remote teaching, or a slight increase in performance in emergency remote teaching. A different result would probably indicate a failed implementation of emergency remote teaching and emphasize the need for careful planning when moving a course online.

2.3. Variables affecting academic performance in emergency remote teaching implementations

In order to find the most relevant variables for analysis, this research turns to the different options available for the design of online courses. Means et al. (2014) and Hodges et al. (2020) define nine dimensions that must be considered in online learning design. Some of these dimensions might not be applicable to emergency remote teaching when campuses are closed –e.g. instruction modality– whereas others cannot be decided upon by the instructors due to the need to comply with pre-existing learning guides –e.g. pacing– or are related to the instructional design and are difficult to be rapidly modified when the course has already started –e.g. role of online assessments, pedagogy or student role online. From the remaining four (student-to instructor ratio, synchrony, instructor role online and source of feedback), this research focuses on the former two, understanding student-to-instructor ratio as class size.

2.3.1. Class size

There is a reasonable agreement in what can be considered small, medium, or large class sizes in online learning. For instance, Hoyt and Lee (2002) and Benton, Li, Brown, Guo, and Sullivan (2015) differentiate between small (10–14 students), medium (15–34 students), large (35–49 students) and very large (over 50 students) classes, while Means et al. (2014) and Hodges et al. (2020) propose four different levels (fewer than 35 students, between 36 and 99, between 100 and 999, and over 1000). Therefore, there seems to be a consensus in that there is a difference between courses with 35 students of fewer, and courses with over 35 students.

In their analysis of a Technology and Education course, Tomei (2004; 2006) found that the ideal class size of the course was smaller in online learning than in face-to-face settings −12 students and 17 students, respectively–, and that online learning demanded at least 20 percent more instructor time and workload than traditional instruction. Later on, based on their review of 20 studies on class size in online learning, Taft, Perkowski, and Martin (2011) argued that class size may depend on the educational framework –constructivist-objectivist continuum, community of inquiry model and Bloom's taxonomy–, but in most stances the optimal number lies in the range from 15 to 30 students; the authors also suggest that classes larger than 30 students would resemble the characteristics of one-way student-instructor communication in traditional settings.

Even though the results of the different studies under analysis do not indicate differences in academic performance, they do agree in that online learning increases the instructor's workload, a factor that must be accounted for in emergency remote teaching because instructors generally require additional time to adapt not only to the shift in their instruction, but also to the characteristics of a new workplace, very likely far less adequate than their offices and classrooms. Finally, Burch (2019) adds that class size may be related to a student's outcomes due to the positive relationship between student participation and outcomes, even though these results must be taken with some caution, as they were only observed in medium and small class sizes –under 30 students– (Parks-Stamm, Zafonte, & Palenque, 2017) and they were also contingent on instructor participation. Based on the above, we would expect to observe worse average academic performance in larger courses.

2.3.2. Synchrony

There are two essential modes of instruction when considering synchrony: synchronous and asynchronous teaching, even though a mix of both may be possible –e.g. Yamagata-Lynch (2014). The difference between the two modes lies in that in synchronous online teaching the instructor and students are physically separated but communicate in real time, whereas in asynchronous online teaching the separation is both spatial and temporal (Roblyer, Freeman, Donaldson, & Maddox, 2007). Examples of the former include the use of videoconferencing software or chats, whereas the latter includes the use of tools such as message boards or pre-recorded videos and presentations, with varying degrees of interactivity. In a way, synchronous courses have a higher resemblance to face-to-face classroom instruction in that students and instructors meet in the same place at the same time, which is not generally the case of asynchronous online learning (Bernard et al., 2004). Asynchronous courses, on the other hand, have other benefits, such as allowing students to have a self-paced approach to the course.

Research on the influence of synchrony in distance or online learning does not seem to agree on which one is more effective when considering student outcomes. Bernard et al. (2004) conducted a meta-review of 232 studies and concluded that, when compared to traditional teaching, asynchronous distance education rendered substantial better outcomes, even though at an expense of higher drop-out rates. However, subsequent studies showed no differences between synchronous and asynchronous teaching delivery modes (Roblyer et al., 2007), pointing out that both types are effective in delivering online teaching, even though students show preference for synchronous course sessions (Skylar, 2009). More recent studies observe differences in higher order thinking skills developed via student social constructivism, in favor of asynchronous teaching (Brierton, Wilson, Kistler, Flowers, & Jones, 2016). The reason for this difference may lie in that asynchronous learning offers more flexibility –self-organization, more time for reflection–, which is why it may yield better results in adult learners (perhaps in combination with optional synchronous sessions), while younger students may benefit more from a structure of required synchronous sessions (Hodges et al., 2020).

2.4. Supporting technologies in emergency remote teaching

The choices in the digital tools available for emergency remote teaching are as varied as the number of possible pedagogical approaches and learning contexts and applications –see Hernandez-de-Menendez and Morales-Menendez (2019) for an overview of current software tools to support educational processes. For every specific aspect of learning, it is very likely that diverse applications are available –as an example, Chaparro-Peláez, Iglesias-Pradas, Rodríguez-Sedano, and Acquila-Natale (2020) evaluate nine different software applications for peer assessment. Before implementing any tool in their pedagogical practice, instructors typically take their time in evaluating the functionalities, operation, installation, and usability of the different range of technologies that may be most effective in their courses. However, in the case of emergency remote teaching, there is little room for testing due to the urgency of moving online in a very short span of time, and therefore instructors tend to turn to what they already know and the tools they have in place before the crisis (Dill, Fischer, McMurtrie, & Supiano, 2020).

In his review of the response to COVID-19 by US and South African universities, Chaka (2020) finds that there are two main types of online tools and resources that have been widely adopted across all institutions: learning management systems (LMS) and video conferencing platforms. Among LMS, Canvas and Blackboard were the most used online tools in the US, and Moodle was predominant in South Africa, which suggests that instructors –and the university at a higher level– chose to resort to their directly available digital platform to support educational processes in the first place. Regarding video conferencing tools, Zoom stands out as the most used tool, followed by other options with collaborative approaches, such as Blackboard Collaborate, Microsoft Teams or WebEx. Despite the prominence gained by these tools during the pandemic, their use in educational settings is not new; for example, McCoy (2015) reports the use of Zoom by doctoral students, and Macaulay and Dyer (2010) detail their experience with the implementation of WebEx in a pilot program to introduce interactive web conferencing in courses at Towson University. Microsoft Teams was launched at the end of 2016, but recent research has already addressed its use in educational settings –e.g., Poston, Apostel, and Richardson (2020).

Other digital tools such as cloud-based file repositories –e.g., Google Drive, Dropbox, Microsoft OneDrive–, messaging platforms –e.g., WhatsApp, Telegram– or social networking sites seem to have had lower adoption rates –even though it is very likely that their use has been commonplace among instructors who were already using them in their teaching and among universities that rely on cloud-based technological infrastructure and applications provided by companies like Microsoft and Google. Based on this evidence, this study analyzes whether the instructors’ choices of digital tools may have any relationship with academic outcomes.

3. Research setting

This study focuses on the changes implemented in the bachelor's degree in Telecommunication Engineering at the School of Telecommunication Engineering (Universidad Politécnica de Madrid). In order to provide contextual background to the reader, this section presents an overview of both the organizational structure of the institution and the degree program to establish a correspondence with the different aspects detailed in the literature review that helps explain the results of the analysis. This section also details the timeline of events to contextualize the decisions made by the instructors during the pandemic and provide further insight about the impact of the implementation of emergency remote teaching at the School of Telecommunication Engineering.

3.1. Organizational structure and overview of the program

Universidad Politécnica de Madrid is a technical-oriented higher education institution that focuses on the different fields of engineering; the only degree programs offered besides engineering bachelor's and master's degree programs are Physical Activity and Sports, and Fashion Design. The university had more than 37,000 graduate and undergraduate students enrolled in official degrees in 2019 (Servicio de Biblioteca, 2020) and is organized around 17 schools and faculties and 10 research centers and institutes across four different campuses. Universidad Politécnica de Madrid operates under a semi-decentralized structure, with all high-level decisions made by the rectorate, while giving a great degree of autonomy to the different schools –an exception to this would be the institution's financial management, which is mostly centralized.

Schools, and more specifically the School of Telecommunication Engineering, are divided into departments; some departments are divided into units spread across different schools. The relationship between departments and schools is similar to that of the university and schools; therefore, departments have some degree of operational autonomy, especially regarding instructional decisions. Bureaucratic tasks for instructors are mostly limited to quality assurance processes. The School of Telecommunication Engineering currently offers three undergraduate and nine graduate degree programs. Each programme establishes coordination mechanisms with regular meetings of course coordinators at programme and year levels, which facilitates the flow of information regarding instructional practices both formally and informally. Business practices are a second source of informal knowledge: a distinctive characteristic of the School, especially when considered at the national level, is the proximity of faculty to leading business companies in the information technology sector, be it for educational collaboration purposes, research projects or supervision of student internships. This proximity facilitates the acquisition of information about how companies develop collaborative and training practices, as well as what software applications instructors can incorporate into their teaching practice.

Technical infrastructure and digital supporting tools for education are provided by the university's Distance Education Bureau, which offers services such as the campus wide LMS (Moodle, which was already used prior to the COVID-19 crisis in the large majority of courses), production of multimedia resources, online learning consulting and virtual labs Gabinete de Tele-Educación, 2020. Other relevant software tools available for all students and instructors include the Microsoft's Office 360 suite and Blackboard Collaborate. In response to the crisis and to prevent system and network overload, an additional instance of Moodle was created only for examination purposes, and Zoom licenses were acquired for the different departments. However, and given the decentralized nature of the university, instructors could choose to use any other technology they deemed convenient for their courses, at the cost of not receiving official support.

The program under analysis in this study is the bachelor's degree in Telecommunication Engineering. It is a four-year degree where the first three years include core courses that are common for all the students; in the fourth year the students specialize in one out of four disciplines. Additionally, elective courses are offered to obtain the necessary credits to complete the studies. All courses are delivered as face-to-face courses. Each academic year 300 new students are admitted in the program, and at any time during the academic year the degree hosts over 1500 students (2017–18: 1528; 2018–19: 1533; 2019–20: 1536). Around 10 percent of the students abandon the program; dropout tends to occur mainly during the first year, and the average time to complete the program is between 5 and 6 years. Due to the technology-intensive nature of the degree, most –if not all– students are proficient in the use of digital technologies, own laptops/tablets and smartphones, and have wireless connection at home, which limits the potential impact of the COVID-19 due to socio-economic differences and the digital divide.

3.2. Timeline of events

In order to get a more nuanced picture of the responses to the pandemic, it is necessary to explain the conditions under which the emergency remote teaching was implemented. In the second semester, courses started on the week of January 29, 2020. By that date, no infection cases had been reported in Spain yet. The first reported case of COVID-19 occurred in La Gomera, Canary Islands, on January 31, 2020 (Linde, 2020); it was not until February 25, 2020, that the first positive case was reported in Madrid (Ministerio de Sanidad, 2020), which prompted the COVID-19 protocol in the region. During the following days, the School faculty started informal discussions about how to better react to a potential pandemic outbreak; at this point, a complete move to online learning had been discarded.

On March 6, 2020, and considering the increasing number of cases in the region, the School Board of Directors sent an e-mail with a notification for an emergency meeting to all course coordinators and student delegates. In the meeting, celebrated on March 9, 2020, different scenarios were considered; a discussion followed about potential courses of action in case of total suspension of face-to-face instruction.

Only one day later, the Government of the Autonomous Community of Madrid published Order 338/2020 (2020), which effectively declared the suspension of face-to-face instruction starting March 11, 2020. In March 14, 2020, the state of alarm was declared in the whole national territory (Real Decreto 463/2020, 2020), resulting in total lockdown of the general population. A partial suppression of the lockdown during the next months, or de-scalation plan, was structured in four different phases (Ministerio de Sanidad - Gobierno de España, 2020). Phase 0 could still be considered an effective lockdown, as the mobility of citizens was heavily limited. Fig. 1 depicts the chronology of events. From the figure, it is straightforward to note the very limited time for reaction that the pandemic left to students, instructors and course coordinators; it is also worth noting that faculty and students of the School of Telecommunication Engineering experienced effective lockdown for the whole duration of the course after the declaration of the state of alarm.

Fig. 1.

Timeline of events (above the horizontal axis, events relative to teaching and learning; below the horizontal axis, events related to regulatory aspects).

4. Method

The study uses a sample of all (N = 43) the courses of the Telecommunication Engineering Bachelor's Degree at the School of Telecommunication Engineering (Universidad Politécnica de Madrid). Two different data sources are used in the study: instructional decisions made on the transition from face-to-face to online learning were collected from an open survey to course coordinators (open and close dates are indicated in Fig. 1); the second data source contains course-level aggregated student grades from the last three academic years (2017-18 to 2019-20). After inspection of the data sets, one of the courses had to be removed because all the teaching and grading was concentrated in the first month of the semester, and therefore no emergency remote teaching was implemented as the classes had already finished by March 14.

The qualitative survey asked course coordinators the following: (a) teaching methods used during in-class and off-class hours, (b) digital tools used to teach the course sessions, (c) number of students regularly following the course, (d) type of assessment activities, (e) tools used for student tracking, (f) likelihood to change the continuous assessment to final-exam only assessment, (g) whether the assessment criteria and/or system were changed, (h) student attendance (class size) during emergency remote teaching, and (i) general comments about emergency remote teaching and main problems encountered. All course coordinators answered the questionnaire.

The statistical methods used to test the differences include one-way repeated measures ANOVA to test for differences in academic performance across the past three years and independent t-tests to test for differences in final grades between courses in the second semester of the academic year 2019–2020 across the following variables from the questionnaire sent to course coordinators: class size, synchrony, and digital tools used by the instructors. For the study, we established two different class sizes (small and medium, under 35 students; large, 36 students or more). A course was considered to be synchronous when it required students to be present and connected at a given time on a given platform for the course session –videoconferencing systems, chats–, and asynchronous when the instructors provided the materials –course documentation, external links, pre-recorded sessions– for learners to study at their own pace.

Finally, the grading data set contains the number of students who achieved a given mark in a course. Because the courses may differ greatly in size, and in order to make comparisons possible, we analyze the relative rates (passing rates and percentage of students achieving a specific grade) when comparing data from different academic years, and yearly variation of passing grades between the previous academic year (2018–19) and the results under emergency remote teaching (2019–20) to observe the effect of the choices in digital tools used as support during emergency remote teaching.

5. Data analysis

Firstly, we use R software (version 4.0.2) to plot the percentage of students passing each course versus the number of students participating in emergency remote teaching, differentiating between elective and non-elective courses (Fig. 2 ). From Fig. 2, every elective course except for two achieve a 100 percent pass rate; another characteristic of elective courses is that they typically may be considered small to medium regarding class size. Therefore, and to better explain the results of the analysis and gain useful insight, the analysis of the global data set will be complemented by separate analyses of the groups of elective and non-elective courses.

Fig. 2.

Number of students vs. percentage of students who passed the course (dot colors represent elective and non-elective courses).

Fig. 3 shows the number of courses according to class size and synchrony delivery type. From the figure, there is a balance between small-medium and large courses, and a slight difference in the delivery mode, with more instructors choosing to adopt synchronous teaching –which, in this case, could be considered as a direct translation of face-to-face content delivery in a virtual space. When electiveness is considered (Fig. 3, right-bottom), the difference in synchrony increases, with almost two thirds of the courses being taught synchronously. However, and as Fig. 4 shows, class size did not seem to determine whether the course was given synchronously or asynchronously, regardless of electiveness.

Fig. 3.

Number of courses by class size and synchrony delivery type (top), refined by course electiveness (bottom).

Fig. 4.

Number of students per synchrony delivery type across elective and non-elective courses.

For the test of differences, we used the R package ggstatsplot (Patil, 2018), which provides support for repeated measures one-way ANOVA and independent t-tests, with the most common options for the analysis –e.g. parametric, non-parametric, adjustment type, report of the results of the analysis, etc.– and combines it with a graphical output. First, we performed the repeated measures ANOVA to test differences across the percentage of students who passed the second semester courses in the past three academic years (sphericity problems were discarded after observation of Mauchly's test results). The result (Fig. 5 ) shows a significant increase (between 7 and 10 percentual points) in the percentage of the students passing the course under emergency remote teaching when compared to the previous two years (no significant differences were found between the 2017–18 and 2018–19 academic years).

Fig. 5.

Differences in percentage of students passing the course in the past three academic years (second semester courses).

When electiveness is accounted for (Fig. 6 ), the analysis reveals no significant differences across elective courses, while the differences are sustained across non-elective (core) courses, which shows that the improvement in academic performance was caused by an overall increase in student outcomes across core courses.

Fig. 6.

Differences in percentage of students passing the course in the past three academic years across core/non-elective (left) and elective (right) courses (second semester courses).

Fig. 7, Fig. 8 further explore the data on a per-grade basis across all courses and non-elective courses, respectively. From the figures, the number of students that had slightly above average or outstanding performance did not change significatively, but the number of students that had a very good performance (from 7 to 10 points out of 10, excluding outstanding students) did vary significantly. The cause of this result might be attributed to an overall shift in individual grades that would cause the usual normal distribution to move toward higher marks.

Fig. 7.

Differences in percentage of students in different grade ranges in the past three academic years (second semester courses).

Fig. 8.

Differences in percentage of students in different grade ranges in the past three academic years in core/non-elective, courses (second semester courses).

To discard potential confounding effects, we also explored the differences across first semester courses in the past three years, all of which were delivered as face-to-face courses. The analysis (Fig. 9 ) shows stability in passing rates (around 80 percent) and no statistical differences across all three years, which suggests that the improvement of passing rates in the second semester could possibly be attributed to the effect of emergency remote teaching.

Fig. 9.

Differences in percentage of students passing the course in the past three academic years (first semester courses).

Regarding the influence of class size and synchrony delivery mode, the independent t-tests (Fig. 10, Fig. 11 ) show no significant differences in the variation of passing grades between emergency remote teaching and face-to-face remote teaching in the previous academic year, and therefore we cannot confirm a possible influence of these variables on academic performance in emergency remote teaching.

Fig. 10.

Differences in the percentage of students passing the course based on class size (left) and synchrony (right) across all courses (top) and only core, non-elective courses (bottom). The percentage reflects the variation, in percentage, of students who passed the course from the 2018–19 to the 2019-20 academic year.

Fig. 11.

Differences in the percentage of students passing the course based on the videoconferencing platform used across all second semester courses.

When considering the different digital tools used as support for emergency remote teaching, all the courses used the institutional LMS (Moodle) as support for content delivery and assignments; two courses did not use any tools other than Moodle. This result emphasizes the importance of having at least some minimum technical infrastructure to support digital remote teaching, especially as a pre-requisite for successful response in emergency remote teaching. Interestingly, and despite its integration with the institutional Moodle, no respondents used Blackboard Collaborate; the introduction of Blackboard Collaborate was relatively new to the university, and therefore it is highly possible that instructors have turned to videoconferencing tools with which they were more familiar, such as Zoom, Skype, Teams or Webex. The results of the analyses suggest that differences in variations of passing grades cannot be attributed to the choice of one software application or another, except in the case of Webex, where the differences are significant. However, this finding should be taken with caution, as only three courses incorporated Webex.

We also explored the data set in search of other potential variables of influence (Fig. 12 ). Passing grade variation across the different years in the degree was considered of special interest, given that we could expect better academic performance among second-year to fourth-year students than among freshmen due to higher experience in the use of the institutional LMS, as well as better organization skills and better communication with instructors. Fig. 12 (left) shows a relatively improved performance associated with more advanced courses; however, the analysis finds that this improvement is not significant. Finally, Fig. 12 (right) shows the variation in passing grades in relation to the instructors' perceived students' attitudes; the results suggest that the improvement was higher in courses with worse perceived students’ attitudes, an unexpected result that should be further explored in the future.

Fig. 12.

Annual variation (from year 2018–19 to year 2019–20) of students passing the course depending on course year (from first to fourth) and students' attitudes (as perceived by the instructors).

6. Discussion

In answering RQ1, we did not observe a special preference for different teaching methods or digital tools, with the exception of the institution's LMS (Moodle, used in all courses) that gave support to course management. Regarding synchrony, nearly 60 percent of coordinators and instructors chose to continue their classes using synchronous teaching, mostly through videoconferencing tools –this percentage rose to two thirds of the courses when only core courses are considered. This result seems to confirm that, when facing punctual changes such as those caused by the pandemic, instructors seem to resort to digital tools that they are most familiar with –i.e., what ‘already works’– and with instructional methods that most easily resemble current practices –i.e., synchronous sessions that mimic face-to-face learning. Of course, time is an important variable to explain these results, as instructors barely had a week to prepare the move to online teaching.

The responses to the open questions in the questionnaire seem to confirm that, even though more than one quarter of the coordinators did not experience the transition as problematic, adaptation time was indeed one of the main problems encountered by the instructors: ten of them complained about the short period of time available to become familiar with the use of new digital tools and the changes in the learning processes. In words of these instructors: “The lack of awareness about all the possibilities and uses of online tools available is being a problem”, or “It is something new and different, and both faculty and students need to adapt. We are just becoming familiar with online teaching tools. Maybe other courses had already worked in this direction, but in our course we still used chalk and blackboard in face-to-face sessions”. Perhaps the most illustrative remark in this regard is that “remote teaching, when properly implemented and planned in advance, may be useful in some instances. In the case at hand, my overall impression is not relevant because there is no choice, we need to adapt. The main problem lies in that the time we had to move from face-to-face to online teaching has been very short”. All these statements emphasize the need for continuous training on the use of digital educational tools and their incorporation to traditional practices as a means to facilitate transition in times of crisis.

Regarding RQ2, the analysis reveals that the overall academic performance of students in emergency remote conditions was significantly better than traditional face-to-face instruction. Our results then seem to confirm, at a larger scale, those of Gonzalez et al. (2020) in a different university in Madrid. However, under that view one could argue that, at least from this experience, emergency remote teaching is a superior form of instruction to traditional face-to-face courses. Of course, this absolute interpretation is probably very far from reality, and it is difficult to think that every higher institution should be in a constant pre-crisis or crisis state to improve their teaching, or that they should just move all their teaching to online spaces.

Further, and answering RQ3, the results would also support that the choice of digital tools, delivery methods or class size does not have any relevance whatsoever in students’ outcomes in remote learning. If so, does it make any difference? What are the underlying causes of the results? What lessons can be learned from this study? While Gonzalez et al. (2020) conclude that the increase in academic performance may derive from an improvement in students learning strategies and self-regulation skills, we aim to go beyond and seek to find alternative explanations from organizational, individual (both of instructors and students) and instruction-related aspects.

6.1. Organizational aspects

The theory revised in our literature review may shed some light in explaining the results of the study from an organizational view. The first thing that must be noted is that the degree of diversification in the activities and processes has been relatively low: the clients served (students) and services delivered (teaching) remained the same, while only the technologies employed changed (and, in some cases, very slightly), which seems to confirm that the more related the activities to the previous ones, the higher the probability of success (Haveman, 1992).

Second, it could be argued that the School of Telecommunication Engineering at Universidad Politécnica de Madrid was relatively well prepared for the crisis in terms of technical infrastructure; a fully functional instance of Moodle has been in place for several years now, and most courses regularly use their Moodle virtual spaces at least as educational content repository and asynchronous communication channel with students via message boards; even though the transition to emergency remote teaching posed some challenges in scaling the system to ensure quality of service for higher number of concurrent users (the LMS gives support to all degrees in the university), additional instances of the LMS were provided to support specific tasks, such as exams.

Third, the existence of formal and informal communication channels facilitated making faster and more informed decisions about the available options, despite the short time available for response. According to Knowles and Saxberg (1988), these informal channels and an organic structure help successfully coping with changes. In this case, flexibility was further enabled by the School's Board of Directors, which established the necessary informal communication channels in early March in anticipation for the crisis, and therefore helped prepare possible responses. It is interesting to observe that this informal discussion was later transformed into formal communication prior to the moment of crisis. Even though the reaction time was too short –as per the statements of one quarter of the instructors–, the fact is that adjustment mechanisms had already been put in place.

Fourth, the federated or semi-decentralized structure also seems to have favored a rapid response: as instructors felt free to decide on which digital tools and what instructional design they implemented in their courses, no time was spent in bureaucracy and compliance with decisions that had to be made at higher levels. In this sense, the results suggest that flexible structures and rich informal information flows, together with a decent technical infrastructure and staff's technical literacy and innovativeness, may help succeed in facing a moment of crisis such as the COVID-19 pandemic. Interestingly enough, later on during the course a notification was sent by the university asking all instructors to only use officially approved institutional software for examinations; in our opinion, this might have had a negative impact on final grades if tools other than the ones mentioned in this study –already approved by the institution– had been adopted by the instructors, which was not the case.

6.2. Individual aspects

6.2.1. Instructors’ digital skills

Based on the results, we must also look into individual aspects that might help explain the findings from this study. A first aspect worth considering is a particular characteristic of the School: its strong technical orientation; because the School specializes in information and communication technologies and systems, most instructors are technology experts and use synchronous/asynchronous communication tools and learning virtual spaces on a daily basis. While we have not found supporting literature on the relationship between instructors’ digital skills and student achievement, particularly in higher education –most of the research on digital literacy of instructors focuses on the development of digital literacy skills or digital competence among pre-service teachers–, it is reasonable to think that it may have been a contributing factor to an effective and rapid deployment of emergency remote teaching. A good example of this is the adaptation of courses with a high workload in laboratory settings; without students being able to physically access the labs, the faculty teaching those courses opted to rapidly develop ad-hoc virtual simulation environments from scratch, something that would have never been possible without said digital skills.

6.2.2. Students’ digital skills and background

On the students’ side, there is prior evidence of the positive relationship between digital skills and academic performance (Kim, Hong, & Song, 2019; Soleymani, 2014). There is an ongoing debate about the fact that being a digital native does not directly equate to being a digital learner –e.g., having developed digital competence in a formal or informal educational setting– (Gallardo-Echenique, Marqués-Molías, Bullen, & Strijbos, 2015), and therefore we cannot make the a priori assumption that students have developed the necessary digital skills to succeed in an e-learning context, or that they have the necessary resources to even follow an online course –the pandemic has unveiled the problems caused by the digital divide in education (Iivari, Sharma, & Ventä-Olkkonen, 2020; Zhong, 2020). However, in our case, that assumption seems reasonable because (1) young people with the highest levels of digital competence tend to be on courses involving ICT and are more favorable predisposed to use digital tools (Sánchez-Caballé, Gisbert-Cervera, & Esteve-Mon, 2021); and (2) most, if not all students are proficient in the use of digital technologies and have their own devices –smartphones, tablets, desktop and/or laptop computers– that they already use in face-to-face courses to take notes and complete their assignments1 . Additionally the degree programme is strongly focused on STEM matters and the majority of students have already specialized in STEM during their secondary education; this focus on STEM matters may have contributed to the improvement in student outcomes, confirming the findings of Vo et al. (2017) in blended learning.

6.2.3. Procrastination and anxiety

Procrastination and anxiety are well-known detrimental variables to academic achievement in online learning, with the former two affecting the latter (Cormack, Eagle, & Davies, 2020; Frazier, Gabriel, Merians, & Lust, 2019; Kim & Nembhard, 2019; Pascoe, Hetrick, & Parker, 2020; Sanchez-Ruiz & El Khoury, 2019). The outbreak of the COVID-19 pandemic may surely have amplified their relevance in the academic achievement of students, but in a more nuanced way than it might seem wherever lockdown has been enforced.

There is enough evidence that the stress experienced by students –and, let us not forget, also by instructors– has increased during the pandemic (Elmer, Mepham, & Stadtfeld, 2020; Son, Hegde, Smith, Wang, & Sasangohar, 2020). The pandemic lockdown stress is also closely related to anxiety, loneliness and depression (Misirlis, Zwaan, & Weber, 2020), and it is therefore a contributing factor in a potential decrease of students’ academic performance. Additionally, different reports have confirmed important changes in consumption habits of Gen Z-ers during lockdown (Hawthorne-Castro, 2020; Jones, 2020), especially in social media, online gaming, and online video and TV/video streaming services, all of which favor procrastination.

We have not tested students’ attitudes and behaviors in this study, and therefore we cannot assess the potential negative impact of these variables, but future research should also consider how the context of the lockdown may have softened their effect. For example, the higher time devoted to digital entertainment may have been compensated by the inexistence of commuting time –on average, between one and two hours in Madrid– and any other social or leisure activities that could not be carried out due to the lockdown. Besides, one of the most usual mechanisms of Gen-Z-ers during the lockdown to cope with stress, anxiety and loneliness has been the use of videoconferencing tools –Zoom being among the most popular, with an increase in use of almost 5000 percent in Spain between March 9 and April 20 (Cuesta, 2020)– to stay in touch with friends and family; a side effect of the wide adoption of these tools is that their use in the courses delivered synchronously may have been perceived as a natural extension of the campus life and face-to-face courses, which might have had a positive effect on learning.

6.3. Aspects related to learning instruction

Some factors relating to the (forced) changes in the instructional design of the courses may also help explain the results of the study. Most coordinators stated that they had to make changes in the different assignments that students needed to complete to pass the course. In the degree, continuous assessment –which comprises multiple individual or team graded assignments and/or tests during the course and, optionally, a final exam– is the default type of assessment, unless students opt for final examination-only assessment; due to the pandemic, many of these continuous assessment assignments were either delayed, simplified –likely reducing the difficulty level– or directly removed.

This decision had two important consequences. First, students have had higher flexibility to take self-paced learning (it was not unusual in face-to-face instruction that students put more effort in preparing some courses than others depending on the due date of intermediate assignments in the different courses). Therefore, students have found themselves in a better position to organize their own study time and pace, including adjusting for the mix of synchronous and asynchronous delivery of the sessions.

Second, assessment activities were constrained by technical, time-related, and even regulatory factors. From the comprehensive map of assessment scenarios in emergency remote teaching by García-Peñalvo, Corell, Abella-García, & Grande, 2020, the available options have been mostly limited to different types of questionnaires, delivery of documents and oral presentations using videoconferencing systems. While many intermediate graded activities may include some of the former, the structure of the typical final exam in the degree2 is very difficult to translate to an online context (Keijzer-de Ruijter & Draaijer, 2019), unless students are allowed to submit a digitized copy of a hand-written exam on paper, which may add technical complexity and be more time-consuming for both students and faculty. Consequently, some courses have turned to multi-choice question tests. These tests may have been perceived as easier by students, and are better suited to assess knowledge rather than skill (Hettiarachchi, Balasooriya, Mor, & Huertas, 2016), which makes it somewhat problematic to compare the results obtained under lockdown with those of previous years.

Finally, we should also consider the possibility of the existence of cheating behaviors. Despite the effort from instructors in taking measures to prevent cheating when designing their exams, the put in place of an institution-wide code of honor for online examinations and the use of plagiarism detection software (Turnitin), proctoring was restricted due to privacy issues. Previous literature supports the idea that students perceive cheating to be easier and more prevalent in online courses, and that unproctored remote exams include more cheating behaviors than proctored ones (Clark, Callam, Paul, Stoltzfus, & Turner, 2020), and therefore we cannot discard the potential effect of dishonest behaviors among some students, a result also observed by Balderas & Caballero-Hernández, 2020 in online exams in a Computer Science and Engineering course during the pandemic in Spain.

6.4. Additional considerations

For a better understanding of the unplanned move to online teaching, we also summarize the course coordinators’ perceptions about the change to emergency remote teaching, focusing on two different aspects: overall perception of emergency remote teaching and main problems encountered in the change process.

6.4.1. Course coordinators’ perception of emergency remote teaching

In general, the overall impression of coordinators about the move to remote teaching under the pandemic is positive (48.8 percent), albeit nuanced. The coordinators find value in online learning, especially from the students' perspective. For example, one instructor states that “[…] students like it. They suggested the use of [Microsoft] Teams, and I know that they are satisfied and have suggested the same to other instructors”; another instructor's comment in this line is that “[emergency remote teaching] is at the same level of acceptance [among students] than offline classes”. Instructors also perceive that students find it useful to be able to revise the content of the session at a later moment, which complements their technical notes.

Other coordinators who have a positive impression also find value in how the move to online teaching has made them reflect about their own teaching. As one coordinator says: “[Online teaching] may be a good complementary tool that may help us reflect about its true value in traditional teaching”. Other coordinators have incorporated this reflection during the implementation of the changes as a result of the first days of their teaching; for example, one coordinator questioned that “Maybe we are heavily leaning toward keeping synchronous learning (in class time), when a good planning of the activities (asynchronous, giving some freedom to students) with a correct control, monitoring and feedback (synchronous in online office hours or asynchronous –correcting and marking, where the [student's] work lies–) may work out very well”. This reflection about synchrony was also shared by other instructors, most of which coincidentally opted for asynchronous delivery modes.

Interestingly, 10 coordinators (23.3 percent) expressed their concerns about student participation and engagement in the course. Comments like “So far, low active engagement of students” or “As of now, I have noted a decrease in participation” illustrate this feeling. However, when observing the data, most of these courses are delivered asynchronously, which suggests that students may prefer to engage actively in synchronous sessions. A possible explanation is that it takes less effort for students to verbally participate in a videoconference than to develop their ideas in writing on an e-mail or message board, with the added benefit of instant feedback in the case of the former. Finally, other 10 coordinators stated that moving to online learning takes time to adapt, and 3 coordinators stated that online learning cannot be a replacement for face-to-face instruction.

6.4.2. Main problems encountered

Twelve coordinators (27.9 percent) did not seem to find any important problems with their adaptation to emergency remote teaching. Among the remaining coordinators, four categories of issues were raised: the first one (10 coordinators, 23.3 percent) groups different objections about the short time required to adapt to the new processes and tools that support emergency remote teaching –e.g., “[Online teaching] requires a training that has a long learning curve and great initial effort”–; the second one (10 coordinators, 23.3 percent) is related to technical problems with the different videoconferencing platforms supporting synchronous sessions –e.g., disconnections, high latency– and with the LMS –maximum file size, uptime and service availability–; the third one (6 coordinators, 14.0 percent) focuses on low student participation, engagement and motivation, including poorer immediate feedback due to lack of visual contact and social presence; finally, the fourth category (3 coordinators, 7.0 percent) relates to a loss in the experimental aspects of learning, which has an impact on courses involving a high amount of laboratory sessions.

7. Conclusion

The present study analyzed the move to emergency remote teaching in all the courses in a bachelor's degree in Engineering and its effects on students' academic performance. The study of the effects of the COVID-19 pandemic in higher education and the implementation of emergency remote teaching has gained interest among scholars, as this special section in Computers in Human Behavior and special issues in other journals evidence –e.g., Reynolds and Chu (2020), and many others under development. Our research study, while arguably limited in scope to one institution and one degree program, has some distinctive characteristics to offer a significant contribution to this new field of knowledge.

From a theoretical approach, the study incorporates organizational aspects, based on the notion of punctuational change in organizational ecology, that may affect successful implementation of emergency remote teaching. The analysis also provides evidence of similar results to those of existing research comparing planned online/blended learning and face-to-face instruction: the findings from this study suggest that class size, the choice of synchronous and asynchronous delivery and the choice of virtual communication tools do not have a significant effect on students’ academic performance.

The study finds that students achieved better results under emergency remote teaching. As mentioned in Section 6, while counter-intuitive, this result confirms, across a larger number of courses, the findings of Gonzalez et al. (2020) in a very similar albeit smaller context. Given that both studies were conducted in a region with strict lockdown during the pandemic, it would be of utmost interest to compare the results with other implementations of emergency remote teaching in regions or countries with less severe lockdowns, or lack thereof.

From a wider perspective, the study seems to be in support of some aspects of the C♭-model for both online and offline environments in higher education (Sailer, Schultz-Pernice, & Fischer, 2020). While the C♭-model formulates a holistic and comprehensive framework that includes proximal and distal factors affecting students' learning outcomes, and therefore its scope exceeds by far the focus of this study, our conceptual framework, the results of this research and the explanations laid out in the discussion section do address some of the foundational blocks of the C♭-model. For instance, the positive results found in this study suggest that distal factors –higher education instructors' knowledge, skills, and attitudes toward technology; their qualification; and institutional, organizational, and administrative factors, together with instructors' and students’ equipment and digital skills– do have an effect on student outcomes.3

The study also offers interesting implications for teaching practice. First, the results suggest that organizational readiness –technical infrastructure and support, flexible structures that facilitate decision-making and empower instructors, the availability of informal communication channels, and development of digital skills of faculty members– have a positive effect when rapidly adapting teaching in the context of a crisis or change of paradigm. Higher education institutions should pay careful attention to these aspects if they seek to be able to quickly respond to environmental changes while sustaining the delivery of high-quality education.

Second, the results highlight that successfully moving to online learning –or, in this case, emergency remote teaching– goes beyond the mere choice of a specific technology. The study did not find significant differences between the different digital tools used in the courses. Currently, the range of software applications to support learning is so wide that instructors might do well valuing compatibility with learners’ –and their own– practices, both in terms of familiarity with the software and its fit with the instructional approach, over other aspects when considering the use of a digital tool.

Third, the choice of delivery mode did not seem to affect students’ academic performance. While this result would suggest that this choice might also be left to instructors, previous literature (Moallem, 2015; Oztok, Zingaro, Brett, & Hewitt, 2013; Xie, Liu, Bhairma, & Shim, 2018) suggests that a mix of both approaches –or bichronous online learning (Martin, Polly, & Ritzhaupt, 2020)– works better because it combines the benefits of both delivery modes –i.e., increased social presence and interactivity in synchronous online learning and self-paced learning and flexibility in asynchronous online learning. In fact, our findings could be the result of the combination of both methods, not within a course but across courses.

Fourth, we found that class size did not have an impact of academic performance. In our literature review, we showed that online learning generally benefits from small- or medium-class sizes; therefore, this result contends previous literature and should be further explored by future research.

7.1. Limitations

This study has certain limitations, of which the specifity of the context stands out as the most notable. As a case study, the results are specific of one institution –and in particular, of one engineering school– and one subject –a bachelor's degree in Telecommunication Engineering-, and the same applies to the conditions experienced during the course –the strictest lockdown among all European countries. The choice of the institution and the subject was made by convenience, and we acknowledge that its effect on organizational aspects –availability of technical infrastructure, organizational structure and processes, students' and instructors' digital skills, equipment and general positive attitudes towards the use of educational technology–, and therefore on the results, is not negligible in the least. In addition, as mentioned in the previous section, the strict lockdown also allows for a very nuanced view of self-regulated learning, especially under emergency remote teaching conditions. The combination of such factors might largely bias the results found in this study; therefore, we do not dare to claim universal validity of our findings, but rather present this case to allow for comparisons with other studies framed in contexts different than the one in this research.

A second limitation has to do with other elements of the C♭-model (Sailer et al., 2020) that are proximal rather than distal to student outcomes, such as the type of learning activities involved –e.g., we only considered delivery mode and supporting technologies rather than the type of learning activity performed–; even regarding student outcomes we just focused on a single aspect: the development of professional and knowledge skills. A more in-depth analysis of the factors affecting student outcomes should adopt a more holistic view of outcomes, including self-regulation, digital skills and attitudes toward digital technology, as well as a more detailed observation of the different learning activities.

A third limitation of the study is the omission of students’ views and perceptions of the process of moving to emergency remote teaching; such a perspective would offer further insight about the different aspects covered in this study and complement those that were left out of the scope of the research.

7.2. Concluding remarks

The COVID-19 pandemic caught the educational world by surprise, forcing higher education institutions to respond with different solutions overnight in a context of unplanned change. A second wave is coming, or has already arrived in some places4 ; many higher education institutions will now extend, adapt or fine-tune their digital processes, and consequently instructors will now extend, adapt or fine-tune their instructional design. Shall it still be considered emergency remote teaching? Until when can this situation be sustainable or considered transitory? If the pandemic has proven something is that unplanned change, even when we find relatively positive results such as the ones in this study, should only be the seed of planned change.

Teaching will definitely change when (instead of if, hopefully) the pandemic is over, and the situation has been a wake-up call to higher education institutions about the need to integrate digital technologies into educational processes. It is time to talk about the digital transformation of education for good, because what may (temporarily) work in emergency remote teaching –e.g., offering a digital copy of the course content, replacing an hour of face-to-face class by a synchronous virtual room using videoconferencing systems, simply sending course materials for students to read, etc.– is definitely not the best way to make the most of the possibilities brought by digital educational technologies (García-Peñalvo et al., 2020).

In the same vein, there is a big difference between emergency remote teaching and a real move to online/blended learning, with the key word here being emergency: all studies being conducted during the pandemic reflect a temporary response from instructors and institutions. It is impossible to sustain a constant state of emergency, and therefore the COVID-19 pandemic should be seen not as a fix before returning to the old ways but as an opportunity to improve digital readiness among higher education institutions. The sudden and temporary state of the changes seen in the delivery of instruction during the pandemic forced instructors to rely on readily available digital tools that facilitated fast adaptation, but a true digital transformation calls for integration of Industry 4.0 tools (artificial intelligence, robots, internet of things, educational data analytics) and rethinking of the teaching-learning process itself (Bonfield, Salter, Longmuir, Benson, & Adachi, 2020; Koul & Nayar, 2020).

Biographies

Santiago Iglesias-Pradas is MSc in Telecommunication Engineering, MBA and PhD in Information Systems by the Universidad Politécnica de Madrid. Santiago is Professor at the School of Telecommunication Engineering (UPM). He focuses his research on e-commerce, technology acceptance and learning analytics.

Ángel Hernández-García is MSc in Telecommunication Engineering, Master SAP in Integrated Information Systems, and PhD in Information Systems by Universidad Politécnica de Madrid (Spain). He is Associate Professor at the Department of Organization Engineering, Business Administration and Statistics (School of Telecommunication Engineering, Universidad Politécnica de Madrid). He focuses his research on electronic commerce, technology acceptance, social media and learning analytics. He has been guest editor and published research articles in leading international journals.

Julián Chaparro-Peláez is PhD in Telecommunication Engineering by Universidad Politécnica de Madrid (Spain) and Professor at the Department of Organization Engineering, Business Administration and Statistics (Universidad Politécnica de Madrid, Spain). His research interests include management information systems, electronic commerce and digital transformation of organizations.

José Luis Prieto is MSc in Physics by Universidad Complutense de Madrid and PhD by Universidad Politécnica de Madrid. He has been Research Associate at the Department of Materials Science, Old Cavendish, in Cambridge University and Associate Professor at York University. He is currently Associate Professor at the Department of Electronic Physics, Electric Engineering and Applied Physics (School of Telecommunication Engineering, Universidad Politécnica de Madrid), Researcher at the Institute for Optoelectronic Systems and Microtechnology (Universidad Politécnica de Madrid) and Deputy Director for Coordination of the Bachelor's Degree in Telecommunication Technologies and Services.