Multilevel IRT models for the analysis of satisfaction for distance learning during the Covid-19 pandemic

Abstract

The Covid-19 pandemic played a relevant role in the diffusion of distance learning alternatives to “traditional” learning based on classroom activities, to allow university students to continue attending lessons during the most severe phases of the pandemic. In such a context, investigating the students' perspective on distance learning provides useful information to stakeholders to improve effective educational strategies, which could be useful also after the end of the emergency to favor the digital transformation in the higher educational setting.

Here we focus on the satisfaction in distance learning for Italian university students. We rely on data comprising students enrolled in various Italian universities, which were inquired about several aspects related to learning distance.

We explicitly take into account the hierarchical nature of data (i.e., students nested in universities) and the latent nature of the variable of interest (i.e., students' learning satisfaction) through a multilevel Item Response Theory model with students' and universities' covariates.

As the main results of our study, we find out that distance learning satisfaction of students: (i) depends on the University where they study; (ii) is affected by some students' socio-demographic characteristics, among which psychological factors related to Covid-19; (iii) is affected by some observable university characteristics.

1. Introduction

Digital transformation is an ongoing process that continually changes the ways people do things, including teaching and learning activities. In the educational context, academic institutions have faced the need for new learning formats more suitable to ensure learning achievements in a globalized society [1]. In this regard, the OECD [2,3] increasingly encouraged the development of digitalized learning environments enabling lifelong learning as framed beyond the boundaries of space and time. In higher education (HE), the creation and diffusion of massive open online courses (MOOCs) as well as the implementation of e-learning and blended education represented relevant attempts to meet this increasing need [4].

Compared to other European countries, Italy showed a huge delay in education digitalization [5]. The common belief that technology-based learning represents a risk to the quality of education caused lateness in the spread of digitalization in Italian HE institutions, confining online courses to certain types of private universities, which are considered providers of lower quality education [6].

The Covid-19 outbreak operated as a turning point in this sense. In March 2020, the Italian Government ordered educational institutions of all grades to suspend their activity as a necessary measure to ensure health protection. Hence, distance learning became the only means to guarantee continuity in education programs, and the Italian education system had to re-organize accordingly. The extensive implementation of technological devices during the Covid-19 emergency forcibly accelerated the progressive Italian HE unfolding to digital innovation, generating a “digital leap” and leading the way towards a new era for HE based on more sustainable approaches [7]. Indeed, before the Covid-19 pandemic, only a few public universities had started implementing platforms to offer distance learning courses [6]. Consequently, when the Covid-19 broke out, teachers and students had to suddenly change their traditional way of teaching and learning. Legislative measures enacted by the Government imposed the development of distance learning solutions, but left every institution to independently organize its activity. Accordingly, most universities exploited one or more collaboration platforms (mainly, Microsoft Teams) to provide lessons, give communications, and perform exams and graduation exams. Some of them also used Moodle to provide asynchronous learning resources (usually, registered lessons and teaching notes) and activities and to perform written exams [6]. Finally, social networks were often adopted to facilitate academic communication.

In this vein, research about distance learning caused by the Covid-19 pandemic plays a significant role in understanding students' and teachers' responses to educational activities conducted via digital devices.

As part of this research line, our study focused on Italian university students' satisfaction in distance learning. The aim was to investigate students' perspective about distance learning facilities compared to place-based classroom learning. Indeed, digitalization serves to enhance universities' products and services; thus, inquiring on students' satisfaction provides hints to fine-tune digital strategies in education, even after the end of the emergency.

The analyzed data comprise students enrolled in different Italian universities. Consequently, a hierarchical structure of the data emerged, with students at the first-level (individual level) of the hierarchy and universities at the second-level (cluster level); thus, a multilevel approach to data analyses was adopted. Indeed, considering that universities may differ regarding their updating and advancement in using technologies [6], it is reasonable to suppose that students' satisfaction in distance learning is more similar for students belonging to the same university than those belonging to different ones.

More specifically, as student satisfaction is an unobservable (i.e., latent) variable, its measurement relies on manifest (i.e., observable) items and, then, we need suitable statistical models for its assessment. In particular, we base our analysis on Item Response Theory (IRT; [[8], [9], [10]]) models exploiting a multilevel structure [11]. We focused on these methods to measure students' satisfaction and, simultaneously, to rank universities according to the satisfaction levels of their students. Thus, variations at both student and university level of the data were considered.

We also investigated the effect of some student and university characteristics adding in the model a set of covariates at the individual and cluster levels, respectively. Individual covariates referred to both socio-demographic (e.g., age, employment), course-related (e.g., type of degree, daily lesson time, digital resources), and psychological factors (i.e., risk perception, stress due to Covid-19, future career anxiety). Moreover, cluster-level covariates included geographical macro area, university size, and type of management (public vs. private).

The remainder of the paper is structured as follows. Section 2 presents a review of the literature regarding the university students' satisfaction with distance learning experience during the Covid-19 pandemic. Section 3 provides a detailed description of the survey and the data, whereas Section 4 focuses on the multilevel IRT model formalization. Section 5 reports the main results and discussion. Finally, some conclusions in Section 6 end the paper.

2. Literature review

Conceived as one of the main objectives of universities, students' satisfaction has assumed a prominent role in the assessment of course effectiveness, improving participation, motivation, learning, and success [[12], [13], [14]].

Driven by the increasing diffusion of technology in education, already before the Covid-19 pandemic, research started to investigate students' e-learning satisfaction and its related factors. In this regard, referring to the existing literature [15] six main dimensions were identifed affecting perceived e-learner satisfaction: (i) Student dimension, including learner attitude toward computers, learner computer anxiety, and internet self-efficacy; (ii) Instructor dimension, referring to instructor attitude toward e-learning and response timeliness; (iii) Course dimension, consisting of course flexibility and quality; (iv) Technology dimension, embracing technology and internet quality; (v) Design dimension, recalling the Technology Acceptance Model (TAM; [16]) that includes perceived usefulness and perceived ease of use factors; (vi) Environment dimension, considering the diversity in assessment and learner perceived interaction with others.

Other studies pointed at instructor interaction, communication, active learning, and efficient assessment of academic progress as determinants of students' satisfaction in online learning [17,18]. Recent research also argued the additional role played by information quality (e.g., accuracy, availability, understandability), service quality (e.g., functionality, facility, navigation), and enjoyment [19,20].

Overall, in addition to the TAM, also the DeLone and McLean's Information Systems success model (D&M; [21]) the Expectation Confirmation Theory (ECT; [22]) and the Technology Satisfaction Model (TSM; [23]) stand out among the most widely theoretical models employed to study the structural relationship between students' satisfaction in e-learning, antecedent factors, and intention to use. Sometimes extended or integrated versions of these models were also considered [13,24].

Nevertheless, although theoretical models helped to explain student satisfaction determinants, research reported that students were still not satisfied as expected [25], highlighting the need for further studies on students' satisfaction in order to gain insights for enhancing technology-based learning experiences.

A recent systematic review of research on online teaching and learning from 2009 to 2018 posed that only 2.75% of studies focused on learner affective characteristics such as satisfaction [26]. This fact highlights that student satisfaction has not yet been thoroughly investigated and convinced about the relevance of increasing studies on this topic. About that, it is a matter of the fact that distance learning amid the Covid-19 pandemic strongly affected students' perception of education quality and their satisfaction [27], moving research to even more new scenarios.

Regarding the studies exploring students' satisfaction in distance learning during the Covid-19 outbreak, several concerns affect specific academic fields particularly damaged by the suspension of lectures and practical training due to the high risk of contagions, such as medicine, dentistry, and nursing (see, for example, [[28], [29], [30]]). Others focused on underdeveloped countries where the lack of technological resources became very challenging for the e-learning implementation, showing that students could not achieve the expected results [[31], [32]]. Finally, another amount of research introduced technology-based practical experiences within a specific context, such as a university lab or degree course. For istance, in [[33], [34]] a study was presented about e-learning in engineering degree programs, highlighting a high overall satisfaction rate but some criticism for distance learning during practical and project work. Specifically, students complained about the lack of exchange between students (and between students and teachers) and the poor responsiveness of teachers while supervising students' work. In the Italian context, examples are reported in [[35], [36]].

It is worth noting that only a few studies exploring students' satisfaction in distance learning during the Covid-19 pandemic included students from more than one university or country [37,38]. Among them, to the best of our knowledge, no study exploited multilevel models for data analysis. In this vein, our study provides new insights on the theme, accounting for variation at both student and university levels and allowing for both student and university variables as predictors of students' satisfaction. Despite not having focused on distance learning amid Covid-19, previous studies highlighted the effectiveness of multilevel modeling in analyzing university students' satisfaction (see, among others, [39,40]).

For students' satisfaction assessment, researchers adopted both qualitative and quantitative approaches. The former involved open-ended questions and interviews, whereas the latter, which is adopted in our study, mainly consisted in conceiving students' satisfaction as a latent construct measured through a set of Likert-type items.

Regarding determinants of students' satisfaction, some research referred to the above-mentioned theoretical models, validating it also in the context of distance learning during the Covid-19 pandemic (e.g., [37]); others focused on a set of characteristics of interest that were hypothesized to affect student satisfaction. For example in [30] students' interests, interactivity, and expectations were investigated. Otherwise, in [78] three types of interaction were examined: student-technology, student-content, and student-student interaction, reporting that only the first two affected students' satisfaction. On the other hand, in [41] a qualitative approach was followed to highlight the strengths and limitations of distance learning and to explore aspects of alternative modes of instruction to be implemented in the long term. Among the distance learning strengths, students cited the feasibility of attending the lectures, the possibility to record the lectures and listen to them again to understand complex topics better, and easier management of their free time and study sessions. On the other hand, students reported among the limitations the internet connection problems, the lack of interaction with both teachers and peers, and the lack of practical training [30].

A further innovative aspect of this study lies in the set of individual-level covariates considered in the model. Indeed, in addition to the socio-demographic and course-related factors, we included psychological variables strongly affecting students' lives during the emergency period caused by Covid-19. Firstly, the pandemic represented a source of stress for university students regarding relationships and academic life (e.g., relationships with colleagues and professors), personal and relational life (e.g., social isolation), and fear of contagion [42]. Indeed, the lockdown protective measure radically changed students' life, moving both formal and informal interactions to online platforms; consequently, negative emotions such as boredom, anxiety, and frustration arose [38,43]. Moreover, several studies showed that students exhibited fear of contagion [44], which increased their levels of stress. Consistently, also personal risk perception affected students' mental health [45] and their responses to the pandemic. In particular, it is reasonable to suppose that students who experienced a higher level of risk perception showed a more positive attitude and satisfaction towards distance learning, considered an effective measure to reduce the risk of contagion.

Finally, the great feeling of uncertainty caused by the spread of the Covid-19 pandemic led students to worry about the long-term effects embracing also their future professional careers [38]. Consequently, we believe that future career anxiety, as well as perceived stress, may influence students' responses to distance learning, impacting their enjoyment and satisfaction.

3. Survey and data

The survey, carried out in 2020, refers to a sample of 985 students from 20 Italian universities. Respondents have been randomly selected using a chain sampling where a number of students enrolled in university associations has been asked to recruit further students among their acquaintances. Students have been contacted and given all the information about the study, and they were asked to participate voluntarily.

The survey was administered during the period November 20, 2020 and December 10, 2020. All the students selected for the study experienced March–May 2020 with a full lockdown for Italians, characterized by massive social restrictions, among which the closure of universities and, thus, 100% distance learning. Selected students were asked to answer referring to this period. Freshmen were not included in the sample. Thus, all respondents involved had the opportunity to assess the difference between the traditional face-to-face learning method (experienced at the university before March 2020) and the distance learning method (experienced in March–May 2020). The analysis refers to an evaluation of 100% distance learning without contemplating the hybrid proposal, which was introduced subsequently.

The next two sections dealt with the description of the sample and the main variables analyzed in the multilevel IRT models (Section 3.1) and a full description of the measures (the psychometric scale) examined in the study (Section 3.2).

3.1. Sample data

The analyzed sample was investigated in relation to several students' and university characteristics belonging to three main domains: subject domain, effectiveness domain, and health domain. Characteristics related to the subject domain of the sample are summarized in Table 1 .

Table 1.

Subject domain - Characteristics of study participants.

Variable	Category	Percentage
Gender	Female	75.74
	Male	24.26
Work	Yes	89.23
	No	10.73
Type of student	Off-site	30.67
	In site or commute	69.33
Degree	Bachelor	61.12
	Master	26.70
	Master Full	12.18
Area (degree programs)	Health Science	10.35
	Scientific	30.35
	Social	32.49
	Humanistic	26.81

Looking at Table 1, we observed that the analyzed sample consisted of 746 women and 239 men, with a combined mean age of 22.10 (SD = 3.36) years. The sample was composed of students enrolled in Health Science (n = 102, 10.35%), Scientific (n = 299, 30.35%), Social (n = 320, 32.49%), and Humanistic (n = 264, 26.81%) degree programs; the majority of them were undergraduate (Bachelor) students (n = 602, 61.12%) and Master students (n = 263, 26.70%); a small percentage of respondents was made up by off-site students (n = 302, 30.67%). Note that, after the Bologna process in 1999, Italy introduced a two-cycle degree structure (the so-called “3 + 2” system), consisting of a three-year degree (Bachelor degree), followed by a two-year degree (Master degree). In addition, there are single-cycle master degrees (Master Full) for some specific courses, such as medicine, law, and architecture.

Details about issues related to the effectiveness domain of universities and distance learning attitudes are in Table 2 .

Table 2.

Effectiveness domain – Students-related distance learning characteristics and universities characteristics.

Variable	Category	Percentage
Distance learning problems	No	56.64
	Yes	43.35
Time spent for distance learning	0	5.38
	1–3 h	19.18
	3–5 h	35.63
	5–7 h	27.41
	7–10 h	10.55
	More than 10 h	1.83
Distance learning resources offered by university	Strongly disagree	9.80
	Disagree	11.60
	Nor disagree nor agree	26.60
	Agree	34.10
	Strongly agree	17.90
Easier management of free time and study sessions thanks to distance learning	Strongly disagree	8.70
	Disagree	15.60
	Nor disagree nor agree	24.90
	Agree	26.40
	Strongly agree	24.40
Distance learning should be integrated to classroom learning	Strongly disagree	12.90
	Disagree	13.00
	Nor disagree nor agree	21.50
	Agree	17.10
	Strongly agree	35.50
University dimension/size	Mega	59.18
	Others	40.81
University location	North	34.11
	Center	11.57
	South	54.31
University type of management	Public	95.13
	Private	4.87

A high percentage of respondents remarked that they had not experienced any problems during distance learning (n = 558, 56.54%), whereas more than one half of students spent between 3 and 7 h attending distance learning classes (n = 540, 63.04%). A high number of students were in mega (i.e., more than 40000 enrolled students) universities (n = 583, 59.18%) with public management (n = 937, 95.13%), and located in the South of Italy (South n = 535, 54.31%; Centre n = 114, 11.57%; Nord n = 336, 34.11%). About half of students stated that their university helped to provide them with distance learning resources and that they managed their free time and study sessions more easily thanks to the introduction of distance learning. The latter information derives from a rating question on a 5-point Likert scale ranging from one (Strongly disagree) to five (Strongly agree).

Further information acquired on students dealt with their psychophysical health conditions (health domain), whose results are summarized in Table 3 .

Table 3.

Health domain - Students psychophysical health conditions.

	Mean	Mode	Median	sd	Range
Stress scale
Social isolation	6.13	7	7	1.82	0 - 8
Academic Life	9.35	9	9	3.46	0 - 16
Fear of Contagion	2.79	3	3	1.01	0 - 4
Perceived risk	3.83	5	4	1.23	1 - 5
Future career anxiety	16.05	20	16	3.62	5 - 20

To assess self-reported health conditions several questions on a 5-point Likert scale ranging from zero (Not at all) to four (Extremely) have been submitted. They were related to the Covid-19 student stress questionnaire introduced in [42] and whose details concerning the analyzed survey are in [46,47]. For the reliability test, Cronbach's Alpha [48] equals to 0.71 was used to assess the internal consistency of the scale.

Among the latent dimensions, we considered in this contribution stress due to social isolation, stress related to academic life, and stress due to fear of contagion. Social isolation and stress related to academic life were obtained by summing up two items for the first dimension (perceived stress concerning social isolation and changes in sexual life) and four items for the second one (stress concerning relationships with relatives, relationships with colleagues, relationships with professors, and academic studying experience). Fear of contagion was also considered to be related to a one single question.

Another aspect we studied was about the perceived risk due to attending in person the courses; it has been acquired by means of an ordinal scale from 1 (Completely disagree) to 5 (Completely agree).

Finally, we assessed students' future career anxiety through 5-point Likert-type items ranging from 1 (Strongly disagree) to 5 (Strongly agree), according to [49]; Cronbach's Alpha was equal to 0.87.

3.2. Measures

The psychometric scale used to measure the student's perspective of distance learning [28] is displayed in Table 4 . The scale consists of twelve items on a 4-point Likert scale ranging from 1 (Strongly disagree) to 4 (Strongly agree).

Table 4.

Distance learning scale.

Measurement items
Preference domain
Clarification sessions is more suitable delivered in distance learning
Assessment is more suitable delivered in distance learning
Effectiveness domain
I did not experience any problems during distance learning
I did not experience stress during distance learning
I had more time to prepare learning materials before group discussion with distance learning
I had more time to review all of the learning materials after class with distance learning
Learning satisfaction domain
Distance learning gives similar learning satisfaction than classroom learning (LS1)
Distance learning could be implemented in the next semester (LS2)
Distance learning gives motivation for self-directed learning and eager to prepare learning materials before group discussion (LS3)
Communication with lecturers and fellow students is easier with distance learning (LS4)
I like distance learning than classroom learning (LS5)
I study more efficiently with distance learning (LS6)

The items are grouped into three sub-scales: i) two items to measure the preference for the distance learning relative to the clarification sessions and assessments (Preference domain); ii) four items to measure the effectiveness of the distance learning, that is, if it represents a problem, a source of stress, or an opportunity (Effectiveness domain); iii) six items (LS1 – LS6) to measure satisfaction with the distance learning (Learning satisfaction domain). Our analysis focused on the learning satisfaction domain, whose frequency distributions of single items are reported in Fig. 1 . For the interpretation of items, note that selecting high response item categories denotes increasing levels of learning satisfaction.

Fig. 1.

Distribution of items measuring the learning satisfaction domain (Top left panel: LS1; Top central panel: LS2; Top right panel: LS3; Bottom left panel: LS4; Bottom central panel: LS5; Bottom right panel: LS6).

Cronbach's alpha of the analyzed dimension is 0.905, whereas the main descriptive indexes concerning each item are displayed in Table 5 .

Table 5.

Learning satisfaction domain – Main statistical descriptive indexes.

	Mean	Mode	Median	sd	Range
LS1	1.87	1	1	0.95	1–4
LS2	2.50	3	3	1.00	1–4
LS3	1.93	2	2	0.85	1–4
LS4	1.88	1	1	0.89	1–4
LS5	1.80	1	1	0.98	1–4
LS6	2.13	2	2	0.99	1–4

Looking at Table 5, we observe that restrictions drastically impaired the possibilities to benefit from living the university life. University students reported growing distress connected to changes in relationships with colleagues and professors and experienced an overall negative judgment about distance learning satisfaction with also increased suffering related to academic studying. Further analyses in Section 5 underline the main finding of our study highlighting the different perspectives depending on universities.

4. Methodology: multilevel IRT model

As outlined in the introduction (Section 1), data at issue present a complex structure characterized by:

• (i)multiple observable items contribute to measure the latent variable of interest (i.e., the student satisfaction for distance learning);
• (ii)hierarchically structured data, with students nested within universities;
• (iii)availability of some exogenous observable student and university characteristics, whose effect on the student satisfaction is of interest.

In the light of these key elements, a suitable statistical modeling approach for the data analysis is represented by the multilevel IRT models. Indeed, the multilevel formulation of an IRT model [11,50] consists of two components that allow us to take into account all the elements above cited:

• 1)a “standard” IRT component for the conditional probability of the observed item responses given the latent satisfaction P(Y| θ) (point i above)
• 2)a regression component for the relation between the latent satisfaction and suitable covariates, P(θ | X , W) (point iii above), taking into account the hierarchical structure of data (point ii above).

In this section, we first illustrate the IRT model component (Section 4.1), referring to the case of ordered polytomously-scored items, and then the multilevel latent regression component (Section 4.2), relying on a formulation based on random effects.

4.1. IRT model component

Let the ordinal outcome Y _ijk be the response on item k (k = 1, …, 6) of student i (i = 1, …, n _j) enrolled in university j (j = 1, …, 20). Each item has m _k (c _k = 1, …, m _k) ordered response categories; in our study all items have 4 response categories, thus m _k = 4 for all k. Moreover, let θ _ij denote the (latent) learning satisfaction of student i in university j responding to the 6 items LS1 to LS6 (see Table 4, Table 5 and Fig. 1). We assume that θ _ij is a random variable having a normal distribution.

Adopting an adjacent-categories approach [51], the model component for the conditional probability of observed item responses Y _ijk relies on the local choice between category c-1 and category c, thus the model can be built from binary IRT models assuming that the binary models simultaneously hold for each pair of response categories (c, c-1) (see [52] for further details). Under this conceptual framework, the conditional probability of student i in university j answering c to item k is given by

$$P\left(Y_{ijk}=c|{ϴ}_{ij},Y_{ijk}\in \left\{c-1,c\right\}\right)=F\left[{\gamma }_k\left({\theta }_{ij}–{\delta }_{ck}\right)\right],c=2,3,4\text{,}$$ (1)

where γ _k and δ _ck are item-specific parameters and F[ ] represents a cumulative probability distribution function. When the standard logistic distribution Λ[ ] is adopted, model in (1) results in the so-called generalized partial credit model [53], which reduces to the Partial Credit Model (PCM; [54,55]) when discrimination parameters γ _k are constrained to 1 for all items

$$\mathrm{P}\left(Y_{ijk}=\mathrm{c}|{ϴ}_{ij},Y_{ijk}\in \left\{\mathrm{c}-1,\mathrm{c}\right\}\right)=\Lambda \left({\theta }_{ij}–{\delta }_{ck}\right),\mathrm{c}=2,3,4$$

Alternatively, the PCM may be formulated in terms of logit as

$$\mathit{log}\frac{P\left(Y_{ijk}=c|{\theta }_{ij}\right)}{P\left(Y_{ijk}=c-1|{\theta }_{ij}\right)}={\theta }_{ij}-{\delta }_{ck},c=2,3,4\text{.}$$ (2)

The logit formulation makes evident how the probability of selecting a higher response category depends positively on the individual latent satisfaction ${\theta }_{ij}$ and negatively on the item-specific parameter ${\delta }_{ck}$. Indeed, parameter δ _ck is usually named item-threshold difficulty and it provides the value of the latent satisfaction θ _ij at which the probability of answering item k by category c equals the probability of answering c-1 (c = 2,3,4). Each item has 4-1 = 3 difficulty parameters, one for each threshold that denotes the choice between consecutive categories (i.e., category 2 vs. 1, category 3 vs. 2, and category 4 vs. 3). Items with high values of δ _ck denote “critical” or “problematic” situations that require high levels of satisfaction to encounter the approval of respondents. On the contrary, items with small values of δ _ck denote situations on which individuals usually agree, independently of their level of satisfaction.

In the PCM we have as many difficulty parameters as the combinations of items and thresholds. A more parsimonious formulation is obtained when suitable constraints on parameters δ _ck are set, as in the rating scale model [56].

Note that the family of adjacent-categories models is much larger because in eq. (1) any strictly monotone distribution function can be used: for instance, the use of the standard normal distribution yields a probit version of the class of adjacent-categories models.

Alternatively to adjacent-categories models, the literature also proposes models based on cumulative logits [57], which yield the graded response model [58] and its special cases, and models based on continuation ratio logits, such as the class of sequential Rasch models [80]. Further discussion on the possible specifications of IRT models for polytomously-ordered items is provided in [59] and specific insights for the multilevel framework are provided in [11].

In this contribution, the preference for the adjacent-categories (logit) models (eq. (1)) and, among these, for the PCM (eq. (2)) was driven by the comparison of different model specifications through the Bayesian Information Criterion (BIC; [60]) index.

4.2. Multilevel latent regression model component

The second component of the multilevel IRT model specifies the relation between students' satisfaction and observable covariates via a latent regression framework accounting for the multilevel structure of data.

Let X and W be two matrices containing the explanatory variables (covariates) at student level and at university level, respectively. Formally, we specify a random-intercept linear model [61,62] for the latent satisfaction.

$${\Theta }_{\mathrm{ij}}={\beta }_{0\mathrm{j}}+{\beta }^{\text{'}}{X}_{ij}+{\mathrm{e}}_{\mathrm{ij}}\text{,}$$ (3)

$${\beta }_{0\mathrm{j}}={\beta }_0+{\tau }^{\text{'}}{W}_j+{\mathrm{u}}_{\mathrm{j}}\text{,}$$ (4)

with β _0j random intercept that accounts for the unobservable effect of university j on the students' satisfaction, β and τ vectors of regression coefficients explaining the effects of student-level and university-level observable covariates on the students' satisfaction, respectively, and e _ij and u _j random effects denoting the error components at student- and university-level, respectively, assumed normally distributed with mean 0 and constant variance. Substituting eq. (4) in eq. (3) the reduced-form of the random-intercept linear model follows:

$${\theta }_{\mathrm{ij}}={\beta }_0+{\beta }^{\text{'}}{X}_{ij}+{\tau }^{\text{'}}{W}_j+u_{\mathit{j}}+e_{\mathrm{ij}}\text{.}$$ (5)

4.3. Model estimation

Substituting eq. (5) in eq. (2) the latent regression multilevel PCM is obtained as follows (c = 2,3,4)

$$\mathit{log}\frac{P\left(Y_{ijk}=c|{\theta }_{ij},X,W\right)}{P\left(Y_{ijk}=c-1|{\theta }_{ij},X,W\right)}={\beta }_0+{\beta }^{\prime }{X}_{ij}+{\tau }^{\prime }{W}_j-{\delta }_{ck}+u_j+e_{ij}\text{,}$$ (6)

Model at issue is characterized by three types of parameters: (i) item-threshold difficulties δ _ck, (ii) regression coefficients in vectors β and τ, (iii) variances of error components u _j and e _ij.

The estimation of parameters is performed along a single step, following several approaches. One of the most commonly used estimation approaches consists of maximizing the marginal log-likelihood function of the model obtained by integrating out the random effects. The resulting function involves a multidimensional integral that may be approximated through numerical integration techniques (e.g., adaptive Gaussian quadrature). For details, see [63], among others. The resulting approximated function may then be maximized with suitable algorithms (e.g., Newton-Raphson algorithm for direct maximization and EM algorithm for indirect maximization). Alternatively, the Markov chain Monte Carlo method can be used [11]; in this case, computing the posterior distributions of the parameters involves high dimensional integrals that can be carried out by Gibbs sampling [64]. In this paper, we consider the maximum marginal likelihood with adaptive Gaussian quadrature and Newton-Raphson iterative algorithm by using the GLLAMM routine by STATA software [65]. Further details are confined in [40].

5. Empirical analysis: results and discussion

In what follows, we describe the results of the estimation of the latent regression multilevel PCM in eq. (6). We first provide details about the model specification. Second, we illustrate the estimates of the item-threshold difficulty parameters (δ _ck) in order to detect the most problematic issues of distance learning. Then, we describe the effects of student- and university-level covariates (β and τ) on the students' satisfaction and we define a ranking of universities based on the assessed second-level random effects (u _j).

5.1. Model specification

As outlined in Section 4.1, the selection of an adjacent-categories model (eq. (1)) with a partial credit specification for item parameters (eq. (2)) was carried out by comparing alternative model parameterizations through the BIC index. This preliminary exploratory phase was provided on models ignoring the multilevel structure of data and the covariates.

After the selection of a PCM, we compared a PCM and a multilevel PCM (both of them without covariates) through a likelihood-ratio test. The statistically significant result of the test (N = 23640, Chi-square = 124.53, p-value < 0.0001) led us to conclude in favor of a multilevel model to take explicitly into account the hierarchical structure of data. The BIC index also led us to the same conclusion (BIC of PCM = 12595.90; BIC of multilevel PCM = 12481.44).

As the last step of the model selection, we estimated the model in eq. (6) adding individual- and cluster-level covariates. At student level, we included age, gender, work condition, student type, type of degree, and area of study (Table 1); we also considered the presence of distance learning problems and resources offered by university, time spent for distance learning, and easier management of free time thanks to distance learning (Table 2) as well as scores on the future career anxiety, on the perceived risk and on the stress scale for social isolation, academic life, and fear of contagion (Table 3). At university level, we considered the size, the geographical area, and the type of management (public vs. private) (Table 2).

5.2. Item-threshold difficulty parameters

The estimates of item-threshold parameters are shown in Table 6 and, to make their interpretation easier, they are also plotted in Fig. 2 .

Table 6.

Estimates of item-threshold difficulty parameters with standard errors, z-value, p-value, and confidence interval at 95%.

Item	Category	Coef.	St. err.	z-value	p-value	95% CI
LS1	2 vs. 1	−3.412	0.345	−9.89	<0.001	[−4.088; −2.736]
	3 vs. 2	0.792	0.261	3.03	0.002	[0.280; 1.304]
	4 vs. 3	1.848	0.272	6.80	<0.001	[1.315; 2.381]
LS2	2 vs. 1	−1.972	0.298	−6.63	<0.001	[−2.555; −1.389]
	3 vs. 2	0.089	0.266	0.33	0.738	[−0.432; 0.610]
	4 vs. 3	0.457	0.264	1.73	0.084	[−0.061; 0.975]
LS3	2 vs. 1	−3.635	0.380	−9.56	<0.001	[−4.380; −2.890]
	3 vs. 2	0.160	0.261	0.61	0.540	[−0.352; 0.671]
	4 vs. 3	2.221	0.271	8.19	<0.001	[1.689; 2.753]
LS4	2 vs. 1	−3.734	0.379	−9.84	<0.001	[−4.477; −2.99]
	3 vs. 2	0.504	0.261	1.93	0.053	[−0.007; 1.016]
	4 vs. 3	1.887	0.270	6.98	<0.001	[1.357; 2.417]
LS5	2 vs. 1	−3.605	0.353	−10.20	<0.001	[−4.297; −2.913]
	3 vs. 2	1.096	0.262	4.19	<0.001	[0.583; 1.609]
	4 vs. 3	1.948	0.275	7.09	<0.001	[1.409; 2.486]
LS6	2 vs. 1	−3.413	0.361	−9.46	<0.001	[−4.120; −2.707]
	3 vs. 2	0.260	0.262	0.99	0.320	[−0.253; 0.774]
	4 vs. 3	1.434	0.267	5.38	<0.001	[0.911; 1.957]

Fig. 2.

Estimates of item-threshold difficulty parameters, by item LS1-LS6 (*: average item difficulty; vertical dashed line: average students' learning satisfaction).

Looking at Fig. 2, we first note that item difficulties range on similar intervals (from about −3.7 for the first threshold to about 2.0 for the third threshold) with the only exception of item LS2 (Distance learning could be implemented in the next semester) that has a much narrower range (from −1.97 for the first threshold to 0.46 for the third threshold). Comparing these results with the distribution of students' learning satisfaction (Fig. 3 ), having a mean equal to 0 and a standard deviation equal to 0.63, we can state that students tend to answer items selecting intermediate categories (i.e., disagree and agree), independently of their level of satisfaction. Extreme opinions are very uncommon and, on average, students agree with the item content: indeed, the average item difficulties (denoted by * in Fig. 2) are around −0.44 for all items, that is, slightly lower than the average learning satisfaction (denoted by the vertical dashed line in Fig. 2).

Fig. 3.

Distribution of students' learning satisfaction.

These results could be interpreted considering that learning activities during the Covid-19 pandemic were configured as an emergency education where the distance modality represented the only instrument to guarantee continuity in educational paths. Many universities had to arrange rapid online and remote lectures and virtual learning environments for the first time, where teachers often merely translated their traditional teaching activities [6]. Thus, it is reasonable to suppose that the implementation of information systems and the organizational matters were at an early stage and have been subsequently improved over time. Therefore, on the one hand, students' recognition of distance learning utility and value during the pandemic probably led them to avoid the extreme negative category (Strongly disagree). On the other hand, the difficulty of universities to fully exploit technological systems for distance learning and the students' discomfort in such rapidly shifting to virtual learning environments [66] may reduce the probability of selecting the extreme positive category (Strongly agree).

Moreover, the items that required a higher level of satisfaction to attain agreement rather than disagreement (difficult items) are those that explicitly compare distance learning with classroom learning (LS1 and LS5). As also outlined in other studies [66,67], students generally prefer classroom learning, even recognizing the advantages of distance learning, when asked to compare them. The main reasons lie in the lack of social interactions and a sense of community, adequate assessment tools, practical training, self-discipline, and concentration (see [30], among the others). However, the integration of classroom learning and distance learning is often seen as an effective solution [68]. Indeed, distance learning also brings some advantages such as the possibility to have open education, greater flexibility, recording and rewatching the lectures, and more time available for other activities [6]. This trend was also confirmed in our study, where more than 50% of the students stated that distance learning should be integrated with in-site lectures (see Table 2).

All these aspects also raise the issue of distance learning sustainability that we evaluated through the item LS2 (Distance learning could be implemented in the next semester). The narrower range of this item also highlights a greater students' propensity to strongly agree with this statement than what they have done with the other items, even for lower latent trait (satisfaction) levels. Several reasons could explain this result: for example, students' being aware of still having the necessity of an education system that ensures health protection, as well as the trust in a promising digitization process that steadily improves people's possibility to reach learning achievement.

5.3. Effect of student and university covariates

Table 7 displays the estimates of elements in vectors β and τ related to the finally selected model (i.e., model with only estimated coefficients significant at 5% level). To make the interpretation of the regression coefficients easier, in the last column of Table 7 the odds ratios (OR) are reported.

Table 7.

Latent regression multilevel PCM: estimates of regression coefficients β and τ, with standard errors, p-values, and OR.

	Coef.	St. err.	z-value	p-value	OR
Type of student: Off-site	−0.160	0.074	−2.144	0.040	0.853
Work: yes	0.325	0.106	3.062	0.004	1.383
Easier management of free time	0.256	0.030	8.646	<0.001	1.292
Distance learning problems: Yes	−0.183	0.068	−2.708	0.010	0.833
Time spent for distance learning	0.120	0.030	3.962	<0.001	1.127
Stress: Social isolation	−0.045	0.019	−2.318	0.027	0.956
Stress: Academic life	−0.056	0.011	−5.107	<0.001	0.945
Perceived risk	0.157	0.028	5.599	<0.001	1.170
University size: Large	0.189	0.087	2.167	0.038	1.208
University location: North	−0.738	0.174	−4.232	<0.001	0.478

It is worth outlining that no significant effect resulted for the following variables belonging to the subjective domain: student's age, gender, type of degree, and area of study. These results revealed no gender differences in students' satisfaction, as also reported in [69], and no lower satisfaction for adult learners who could be disadvantaged due to their less competence in digital technologies [70]. Moreover, our findings pointed at no differences according to the area of study, not supporting the particular concern argued in the literature about the health science field [30]. Concerning the type of degree, a not significant effect on students' satisfaction was also reported in [71].

Also, the anxiety for the future career and the stress due to the fear of contagion, even if recorded as widespread feelings among the involved students in line with other studies [38,44], did not significantly affect students' distance learning satisfaction. Among the university characteristics, no significant difference resulted between public and private universities.

Looking at Table 7, a negative impact on the learning satisfaction was observed in relation to the location of the university, with students whose university was located in the North of Italy strongly less satisfied than their colleagues studying in the Centre or in the South (OR = 0.48). A negative impact was also observed for students that had problems with distance learning (OR = 0.83), students studying off-site (vs. students in-site; OR = 0.85), and, even if at a lower extent, students with stress due to social isolation (OR = 0.96) and with stress due to academic life in remote (OR = 0.95). On the opposite, condition of working students (OR = 1.38), devoting more time to distance learning (OR = 1.13), easily managing the free time thanks to distance learning (OR = 1.29), having a high perceived risk to Covid-19 contagion (OR = 1.17), and studying in large universities (OR = 1.21) contributed to a positive impact on the learning satisfaction.

Regarding individual-level covariates, many of our findings find support in the literature. Distance learning is preferred by working students who particularly appreciate the flexibility and the absence of time and space restrictions [72], allowing easier management of working, studying, and free time. As our results outlined, the latter aspect improves students' satisfaction with distance learning, considered one of its principal strengths [30]. A further aspect appreciated by students is the possibility of attending lectures at home without going to the university, especially for those who remained to study in their region of residence [73]. Indeed, the latter probably travel more than off-site students to reach university, which could explain their higher level of satisfaction with distance learning. About course-related factors, according to other studies [30,71], our results showed that having distance learning problems decreases students' satisfaction, whereas more time spent on distance learning per day increases the satisfaction, probably reflecting greater student engagement. Finally, the significant effect of stress and risk perception was consistent with previous studies addressing the psychological consequences of the Covid-19 pandemic on students' lives and responses to distance learning [38,45,66,71].

Regarding the cluster-level covariates, we can offer some speculations. Firstly, we found that students enrolled in northern universities were less satisfied. This inequality could lie in the different learning environments, technologies, and services implemented. About that, a survey launched by the Italian National Agency for the Evaluation of the University System and Research (ANVUR) aiming at evaluating distance learning facilities in the Italian universities highlighted some differences according to the geographical macro-areas (https://www.anvur.it/attivita/ava/didattica-a-distanza/). In particular, the questionnaire addressed to Rectors/Directors investigated the universities' endowment of digital teaching technologies before the emergency (question 1) and the type of digital teaching technologies the universities had during the emergency (question 3), among the others. The 78% of Rectors of northern universities affirmed that their university was equipped with technologies for distance learning even before the Covid-19 emergency compared to the higher response rate (91%) of Rectors of southern and central universities. Moreover, northern universities were less inclined to improve their e-learning platforms at the beginning of the pandemic (65% vs. 73%), as well as provide students with platforms for creating and sharing teaching material and tests, such as Google Classroom (70% vs. 85%), teacher/student online chat rooms (39% vs. 54%), and MOOCs (13% vs. 18%). On the other hand, northern universities used more video-recorded lessons (65% vs. 48%). Similar percentages were found for live streaming lessons (78% vs. 82%). In summary, a greater attention by southern and central universities emerged in providing students with interaction opportunities, which revealed to be one of the most significant factors affecting student satisfaction in online environments [74].

However, we cannot exclude that the macro-area difference depends on other factors, such as the uneven impact of the pandemic in the different Italian regions, especially at the beginning of the pandemic emergency (March 2020). In this scenario, northern universities had to cope with emergency learning implementation in a context more overwhelmed by the effects of the pandemic.

5.4. University ranking

A useful output characterizing multilevel models is given by the second-level random effects u _j. This error component captures that part of heterogeneity in the learning satisfaction that is not explained by the observed variables and is due to unobserved characteristics at university level (e.g., organizational aspects of teaching activities that were not observed). In brief, it represents the latent effect of the university on the learning satisfaction of all its students.

Fig. 4 displays the caterpillar plot with the ranking of the 20 universities according to point estimates of u _j (j = 1, …, 20): increasing values of u _j denote increasing levels of learning satisfaction, being constant the observed covariates. Around each point estimate of u _j, Fig. 4 shows the related confidence interval at 95%.

Fig. 4.

Caterpillar plot: estimated second-level random effects with confidence interval at 95%, by university.

Looking at the caterpillar plot, we note that almost all the confidence intervals overlap with the exception of the universities at the two extremes (see intervals for universities 1 to 4 vs. intervals for universities 17 to 20). This denotes a latent effect of universities 1 to 4 on the learning satisfaction that is significantly worse with respect to the latent effect of universities 17 to 20. In particular, the top four universities are the University Suor Orsola Benincasa of Naples, the University Milano-Bicocca, the University of Turin, and the University of Bologna. On the other hand, the bottom four universities are the University of Naples “L'Orientale”, the University of Salerno, the University of Florence, and the LUISS University.

Our results were partially in agreement with the StuDocu World University Ranking 2020 (https://www.studocu.com/en-gb/world-university-ranking/2020/EMEA) considering the dimension “Studying Remotely”, namely the facilities offered by universities to follow classes online and the easiness to contact teachers/professors remotely. Indeed, the University of Naples “L'Orientale”, the University of Salerno, and the University of Florence reached a lower evaluation than the University Suor Orsola Benincasa of Naples, the University Milano-Bicocca, and the University of Bologna. Conversely, different results were obtained for the University of Turin and the LUISS University.

From a practical perspective, it would be useful to provide policymakers of universities 1 to 4 with insights about the organizational aspects adopted by universities 17 to 20 in order to improve the distance teaching and, thus, to increase the satisfaction of students.

6. Conclusions

Aware that digital developments in education due to the Covid-19 pandemic have represented a point to restart from rather than come back to, our study provided interesting hints on students' satisfaction with distance learning and its related factors. Both policymakers and HE institutions could benefit from the results, shedding light on what most needs to be changed or improved to foster a new and more effective way to teach in the future. In this regard, the combination of traditional and technology-based teaching methodologies offered by blended education appears as the favorite direction, finding heavy support from students. Thus, studies addressed students' perspectives about distance learning, as ours, allow us to detect suggestions to fine-tune educational strategies according to students' preferences.

Compared to other studies exploring university students' satisfaction in distance learning, our study presents some important strengths. First, the data comprise students enrolled in various Italian universities, providing insights into differences between universities. Second, a multilevel Item Response Theory approach to data analyses was adopted, giving information about student and university variability. Third, both student- and university-level variables were considered predictors of students' satisfaction; the former also accounted for psychological variables that strongly affected students' lives during the Covid-19 pandemic. As the main limit, the study relies on non-probability sampling, thus conclusions cannot be easily generalized to the entire population of Italian academic students.

An interesting route for future work is to replicate the survey to explore students' satisfaction with blended learning, combining distance learning with classroom learning. This type of learning approach has been adopted in the Italian universities in the second phase of the pandemic emergency (September 2021–June 2022) to face the age-old problem of overcrowded classrooms: in order to guarantee the distancing among students, the classroom capacity has been reduced, thus lessons are nowadays attended, at the same time, by some students in site (classroom learning) and by the remaining students in streaming (distance learning). This type of learning approach, which could become the “standard” approach in the short-medium time, raises new challenges for academic teachers, involving the use of digital techniques and the best educational strategies to involve both types of students (i.e., students that are in the classroom and students following in streaming). Factors affecting students' performance in online and blended education environments could also be considered for developing proper teaching strategies [75].

Another promising challenging research line is based on the evaluation of distance learning through the recognition of satisfaction induced by students' facial expressions during learning. It introduces the assessment of distance learning in the context of artificial intelligence. The analysis can make use of recent developments introduced in works such as by [[77], [79]], with a possible implementation of techniques already used in other experimental contexts, as pointed out in [76].

Author statement

Silvia Bacci: Conceptualization, Methodology, Software, Supervision, Visualization, Writing – original draft, Writing – review & editing; Rosa Fabbricatore: Conceptualization, Data curation, Formal analysis, Validation, Writing – original draft, Writing – review & editing; Maria Iannario: Conceptualization, Data curation, Investigation, Methodology, Supervision, Writing – original draft, Writing – review & editing.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Biographies

Silvia Bacci, PhD, is Associate Professor of Statistics at the University of Florence. Her research interests are methodological developments in the fields of latent variable models with a special focus on latent class models, multilevel models, item response theory models, and models for longitudinal and multilevel data; applications are mainly oriented to the educational field.

Rosa Fabbricatore is a psychologist and a PhD student in Social Sciences and Statistics at University of Naples Federico II, Italy. Her research interests focus on statistical methods applied to social sciences, educational and social health psychology. In particular, her main research topics relate to latent variable models such as structural equation models (SEM), Item Response Theory (IRT) models, and Gaussian mixture models (GMM).

Maria Iannario, PhD, is Associate Professor of Statistics at the University of Naples Federico II. Her research interests focus on methodological developments in the field of ordinal categorical data. She introduced uncertainty mixture models and extended them to nonstandard situations as in multilevel/hierarchical contexts and when overdispersion or shelter effects are present. She has coordinated several projects and has been awarded with grants with an international profile as a Fulbright scholar.

Data availability

Data will be made available on request.