Abstract
Protective factors that build students’ resilience are known. A six-week digital resilience training program was developed on the basis of theory, evidence, and contextual information. The feasibility study sought to evaluate the acceptability, appropriateness, demand, implementation, and limited efficacy of a digital resilience skills enhancement program for undergraduate students. A single group, pre-test, post-test, concurrent mixed methods design among 10 undergraduate students was conducted in one university in Singapore. The content analysis concluded that students accepted and perceived the digital resilience skills enhancement program as appropriate. Students also proposed several improvements, such as the initiation of the program and revisions to the content. The Wilcoxon signed-rank test found significant improvements in resilience (p = 0.02) and meta-cognitive self-regulation (p = 0.01) scores with medium (d = 0.79, 95% CI: −0.15 to 1.74) and very large effect sizes (d = 1.31, 95% CI: 0.30–2.33), respectively. Students found the digital resilience program appropriate and were able to apply their newly acquired skills to promote their resilience and learning. Although, several improvements are proposed to enhance the rigor of the digital resilience program, the findings of this study suggests that digital resilience programs are important for students’ well-being.
Keywords: resilience, undergraduate students, mixed methods, feasibility study
1. Introduction
The concept of resilience has shifted since its inception. The initial classification of resilience has been associated with individual traits comprising intrinsic factors, such as coping skills and being positive [1]. With the expansion of the resilience research, extrinsic factors such as relationships with others and resources within the environment positively influencing one’s resilience are increasingly acknowledged [1,2]. Contemporary resilience scholars now view resilience as a process, where protective factors (e.g., coping or resources) play an unequivocal role in inoculating an individual against psychological distresses [2,3].
In line with the contemporary theoretical stance on resilience, this study adopts the definition by Van Breda [4] that is “the multilevel processes that systems engage in to obtain better-than-expected outcomes in the face or wake of adversity” [4]. According to Van Breda [4], multilevel processes refer to various mechanisms that may occur at the individual (e.g., positive mindset) or environmental level (e.g., community resources) in which resilience is fostered. Systems refer to the ‘unit’ that is engaging in these processes and can be scaled from the cellular (e.g., physiological response to stress) to the societal level (e.g., interactions with other individuals in the community) [4].
A better-than-expected outcome refers to a contextualized range of outcomes (e.g., resistance) across a population [5,6]. This meant that a considerably good outcome is determined by the individual within a certain context. This allows the individual to determine what constitutes a positive outcome in the wake of adversity [4]. This proposes that resilience can be developed and enhanced.
In spite of the undisputable benefits of resilience in students’ higher education (HE) journey, a meta-analysis reports a global prevalence of 36% for low resilience among HE students [7]. Students with lower resilience are associated with poorer well-being manifested through high level of stress, drop out from school, and poor performances in coursework [8,9]. Higher levels of stress can result in negative impacts on students’ learning outcomes in several ways [10,11]. Studies have shown that students experiencing high levels of stress have reduced pleasure in studying [12] and impaired memory retrieval [13]. When students are unable to learn properly, they may experience greater stress and anxiety [10,11]. Considering that being resilient can have protective effects against stress and mental health [9], it highlights the direct and indirect roles that resilience plays in supporting students’ learning and stress outcomes.
In addition, the presence and persistent aftermath effects of the coronavirus disease 2019 (COVID-19) pandemic led to heightened mental health issues among HE students [14,15]. Given the magnitude of the potential downstream implications, a resilience training program must be developed to promote HE students’ mental well-being, learning outcomes, and academic success.
The interest in resilience training and its impacts is considerable. Two recent meta-analyses [16,17] positioned resilience training as a potential universal prevention strategy that should be incorporated into the formal university curriculum. Specifically, the review concludes that students’ resilience and mental well-being (e.g., depressive and stress symptoms) improve following resilience training [16]. Although resilience is described to be influenced by personal, relational, and environmental protective factors, these are not consistently incorporated into the existing resilience training. This calls for a closer examination of how resilience training can incorporate personal, relational, and environmental factors.
With greater advancements in the technological infrastructure and need for remote learning due to the COVID-19 pandemic, institutes of higher learning (IHL) have transited to the digital platform as a medium of instruction [18,19]. Although the use of an asynchronous online learning platform potentially addresses the cost and scalability issues, drawbacks include the absence of institutional support and poor engagements [20]. Substantial thought about the curriculum design with a combination of learner-centered activities (e.g., quizzes and forums) is needed to engage learners [21].
On the contrary, blended learning considers the use of synchronous (e.g., traditional lectures) and asynchronous engagement tools (e.g., forum discussions) and integrates the merits of face-to-face and online learning [22]. However, it is resource-intensive and more challenging to implement compared with a solely asynchronous online learning approach.
Considering the scarcity of resources, preliminary evaluations using a feasibility study design are useful to ensure that a newly curated digital resilience training program is appropriate and sustainable [23]. The guidelines [16] for developing a feasibility study comprise the following domains: (1) acceptability (i.e., appropriateness or suitability), (2) demand (i.e., actual use), (3) implementation (i.e., facilitators or barriers), (4) practicability (i.e., ability to carry out activities), (4) adaptability (i.e., work in a new platform), (5) integration (i.e., fit in the infrastructure), (6) expansion (i.e., ability to fit in the organization), and (7) limited efficacy testing (i.e., intended effects).
To the authors’ knowledge, few studies examine the acceptability, demand, practicability, and treatment dose effect. In addition, resilience interventions thus far are developed and evaluated in other contexts and countries. Given that the environment plays an important part in developing resilience; developing and evaluating a context-specific resilience intervention are important. Therefore, this study seeks to develop and evaluate a digital resilience skills enhancement (RISE) program for undergraduate students using a feasibility study design.
2. Methods
2.1. Design
A single-arm, pre-test, post-test feasibility study was conducted from October to November 2021. A concurrent mixed-methods approach was used to assess the acceptability, appropriateness, demand and actual use, and limited-efficacy testing of the digital RISE program [23]. The qualitative component explored students’ acceptability, appropriateness, demand, and actual use, whereas the quantitative approach was used to examine the usage patterns and limited efficacy testing on resilience, social support, and learning scores. This study was prospectively registered on the clinical trials register (NCT05072340).
2.2. Participants and Setting
The digital RISE program was offered to all full-time undergraduate students in one university in Singapore. Advertisement posters were disseminated through the university’s online learning and social media platforms. A convenience sampling approach was used to recruit students. Participants were eligible if they were a full-time undergraduate, above 18 years old, could comprehend the English language, and had a device that could connect to the Internet. Students with any self-reported medical history of any mental health conditions or that received similar psychosocial interventions were excluded from this study.
2.3. Development of the Digital RISE Program
The digital RISE program was developed using a three-step approach for the development of complex interventions with reference to Medical Research Council guidance [24]. First, a theoretical basis was formed to explain the mechanism of the intervention. Second, evidence was gathered to deepen the understanding of the effects of resilience training. Finally, to ensure that the intervention is contextually relevant and user-centered, key stakeholders’ perception and preferences for receiving resilience training were solicited. Figure 1 presents the schematic of the development of the digital RISE program.
Figure 1.
Overview of the development and evaluation process of RISE.
2.3.1. Theory
The society-to-cell resilience theory [25] and the transactional model of stress and coping [26] were used to provide the theoretical basis. Resilience theory proposes that resilience may be enhanced through several protective factors [25]. The protective factors include personal (e.g., coping abilities), relational (e.g., presence of secure social networks), and environmental aspects (e.g., availability of resources).
Given that resilience may be developed after experiencing adversity, the transactional model of stress and coping [26] was used to map the causal pathway of how the RISE program may work. With these two as bases, this study postulated that, when a resilient individual experiences stress, they can rely on resilience protective factors to overcome the challenges.
2.3.2. Literature Evidence
Evidence was gathered by conducting three systematic reviews [16,17,27]. First, the meta-ethnography of qualitative studies concluded that a higher students’ resilience is influenced by personal, relational, and environmental factors [27]. Beyond the known personal and relational factors [27,28], it highlights the importance of the IHLs in shaping students’ resilience, suggesting that resilience skills should encompass the introduction of school-based services [19].
Second, the meta-analysis [16] further highlights the role of relationships in enhancing students’ resilience and proposes that resilience training should comprise skills that build social competencies. Two systematic reviews were conducted to identify the features of resilience programs that could best promote resilience [16,17]. The findings proposed that the resilience program should comprise of lectures and discussions delivered over the Internet and adopt a flexible schedule [16,17].
2.3.3. Contextual Information
As part of the development process, a qualitative study was conducted to ensure contextual relevance to the RISE program [28]. The study gathered students’ suggestions and preferences for receiving resilience training and provided insight into the content development of the RISE program. From a design perspective, students proposed the use of blended learning pedagogy and short videos. In addition, students recommended using alternative forms of information delivery, such as narratives from resilient individuals and utilizing reflective practices. They also proposed that the RISE program should be introduced during the school term, right before major assessments, to allow students to apply their newly acquired skills [22].
2.4. Digital RISE Program
The information gathered from the theories, evidence (i.e., systematic reviews), and contextual information (e.g., qualitative study) was consolidated and mapped. The digital RISE program was designed as six weekly sessions that were hosted on the university’s online learning platform (Figure 2) and made available through a mobile application. The six sessions included: (1) introduction to resilience and embracing change, (2) coping strategies, (3) creating positivity, (4) shifting mindsets, (5) building social competency, and (6) preparing for the future. The details of each session are described in Table 1.
Figure 2.
Web interface of the RISE program through the online learning platform.
Table 1.
Topics, session components, learning activities, and take-home tasks.
| Session (Duration) |
Aims | Rationale | Topics | Learning Activities | Learning Outcomes | ||
|---|---|---|---|---|---|---|---|
| Preparatory | Online | FTF/Discussions | |||||
| Resilience and embracing change (78 min) |
Present concept of resilience and readiness to incorporate resilience enhancing strategies |
|
|
|
|
Virtual face-to-face via Zoom |
|
| Coping strategies (28 min) |
Introduce personal, relational, and environmental coping strategies |
|
|
|
|
Forum |
|
| Creating positivity (22 min) |
Impart strategies to promote positivity |
|
|
|
|
Forum |
|
| Shifting mindsets (23 min) |
Inculcating a growth mindset |
|
|
|
|
Forum |
|
| Building social competency (84 min) |
Improve awareness of communication and social competencies skills |
|
|
|
|
Virtual face-to-face via Zoom |
|
| Preparing for the future (128 min) |
Provide insights towards career and finances |
|
|
|
|
Virtual face-to-face via Zoom |
|
FTF: Face-to-Face.
The digital RISE program adopted the blended learning framework [29]. In each week, students were provided access to review the online resources (e.g., reading materials and videos) at their own pace. Online videos comprised traditional lectures or narratives in the form of interviews with invited content experts. In light of the COVID-19 pandemic and numerous competing priorities, physical sessions were replicated in a virtual setting and were held remotely via virtual face-to-face discussions through Zoom (Weeks 1, 5, and 6) and forums (Weeks 2, 3, and 4). At the end of each week, participants were provided with take home tasks and quizzes that aimed to consolidate and evaluate their learning outcomes. Weekly reminders were sent via the learning platform’s announcement function and emails. The total possible duration (inclusive of videos and virtual discussions) for the RISE program was 365.89 min.
2.5. Content Validation
Content validation was conducted to assess the quality and accuracy of the RISE program. The health-related website evaluation tool [30] comprised 36 items across 8 domains, namely (1) content, (2) accuracy, (3) author, (4) currency, (5) audience, (6) navigation, (7) external links, and (8) structure. The items were rated agree (2 points), disagree (1 point), or not applicable. The scores were summed and the quality was assessed based on the following cut-off scores: poor (<75%), adequate (75%), or excellent (at least 90%). A total of five content experts, specifically an educator, student support manager, mental health practitioner, executive coach, and researcher, were invited to validate the contents (Table S1). The scores ranged from 94% to 100%, suggesting the excellence of the RISE program [30].
2.6. Data Collection
Students were recruited from September to October 2021. Students interested in the study expressed their interest through a link provided in the advertisement posters. The primary researcher approached the interested participants and provided them with more information. All clarifications were addressed prior to the obtained informed consent. Participants completed the pretest measures prior to the commencement of the RISE program in October 2021. Following the conclusion of the intervention, students were invited to the complete the post-test in November 2021. The survey comprised a sociodemographic form and three outcome measures (resilience, social support, and learning). The qualitative data were collected via email correspondence. Participants were provided with a list of open-ended questions and given time to respond. Participants were provided with a remuneration of 10 SGD for their participation.
2.7. Outcome Measures
The quantitative measures were collected using Qualtrics and comprised the following domains: (1) sociodemographic data, (2) resilience, (3) social support, and (4) learning. The sociodemographic, academic, and economic characteristics of the students were solicited. Open-ended questions were developed with reference to the guidelines of Bowen and Kreuter [16] for feasibility studies.
2.7.1. Resilience
Resilience was measured using the Connor-Davidson Resilience scale that consists of 25 self-rated items (CD-RISC-25) [31]. The items were rated from “not true at all” (0) to “true nearly all the time” (4), with a higher score suggesting greater resilience. The CD-RISC has the best psychometric properties when compared with other measures of resilience [32], with a Cronbach’s alpha of 0.89 [31].
2.7.2. Social Support
Social support was assessed using the Multidimensional Scale of Perceived Social Support (MSPSS) [33]. The MSPSS comprised 12 items and assessed students’ social support from three sources, namely family, friends, and significant others. The items were rated on a 7-point Likert scale from “very strongly disagree” (1) to “very strongly agree’ (7), with a higher greater score proposing that students have greater access to social support [33]. The MSPSS demonstrates good psychometric properties with a Cronbach’s alpha ranging from 0.84 to 0.92 [34,35].
2.7.3. Learning
Motivated Strategies for Learning Questionnaire (MSLQ) [36] was used to examine the effect of the RISE program on students’ motivational orientations and learning outcomes. MSLQ consisted of 81 items across two domains: (1) a cognitive view of motivation and (2) learning strategies [30]. This study adopted three subscales, each from the motivation and learning strategies domains. The three subscales from the motivation domain were the control of learning beliefs, self-efficacy for learning performances, and test anxiety. The three subscales from the learning domain consist of meta-cognitive self-regulation, time and study environment, and effort regulation. The items were rated on a 7-point Likert scale, ranging from “not true at all of me” (1) to “very true of me” (7), with higher scores indicating higher motivation or the better use of learning resources [36]. MSLQ has sound psychometric properties [36,37].
2.7.4. Participants’ Acceptance and Usage Patterns
Open-ended questions were curated to explore participants’ acceptability, perceived appropriateness, and actual usage of the intervention (Table S2). The usage patterns comprised data such as participants’ access time, viewing frequency, duration, and completion rate. The collected data were used to address numerous aspects of feasibility testing and to corroborate participant’s acceptance, assess the demand, and review the implementation of RISE.
2.8. Data Analysis
2.8.1. Quantitative Analysis
Stata version 17 [38] was used for the data analysis. All survey data were deidentified prior to analysis. Descriptive statistics, such as the median and interquartile ranges, were used to report continuous variables, whereas the frequency and percentage were used to present categorical variables. The Wilcoxon signed-ranked test was used to compare the median and interquartile ranges across two time points. Box plots were used for graphical representations. Effect sizes were used to measure the magnitude of the differences between the means at the pretest and post-test [39]. Effect sizes were computed as Cohen’s d with the following cut-offs: small (d = 0.2), medium (d = 0.5), large (d = 0.8), and very large (d =1.3) [39].
2.8.2. Qualitative Analysis
A content analysis [40] was used to analyze the qualitative data. First, two researchers (DA and LY) independently reviewed the participants responses to the open-ended questions to increase their familiarity with the data. Second, the authors used different colored highlighters to perform manual coding independently. The codes were exported into a tabular format using Microsoft Excel. Third, the two sets of codebooks were brought together for comparison, and discussions were frequently held to resolve all discrepancies. Inter-rater agreements were calculated using Cohen’s Kappa, κ, with a value of 1 suggesting perfect agreement to −1 as the absence of agreement [41]. Values above 0.75 were considered excellent in agreement, while values between 0.60 and 0.75 were considered a good agreement [41]. The inter-rater agreement of the selected codes was 0.64, suggesting a good agreement. A third author (JC) was brought in for further deliberations. Finally, the categories were developed independently and compared for cohesiveness.
2.8.3. Mixed Data Analysis
The concurrent mixed data analysis approach with seven steps [42] was undertaken to integrate the quantitative and qualitative data. The first step involved the display of quantitative and qualitative data. Second, the respective datasets were reduced. Third, the quantitative data (e.g., results of CD-RISC, MSPSS, and MSLQ) were qualitatively transformed into narratives. The content analysis of the qualitative data documented the frequencies of the categories provided by each student, which transformed the qualitative data into quantitative data. In steps four to six, both sets of data were consolidated, compared, and used to explain both sets of findings. Finally, the data were integrated into one coherent outcome.
2.9. Ethical Considerations
This study conformed to the Declaration of Helsinki. Ethical approval was obtained from the university’s institutional review board (NUS-IRB-2021-594) prior to the commencement of this study. Participants were informed of the study’s goals and were provided opportunities to clarify their doubts. Written informed consent was obtained from all participants.
3. Results
3.1. Participant Characteristics
A total of 43 students were assessed for eligibility. Fifteen students did not meet the inclusion criteria, and 18 students decided not to participate. An eventual sample of 10 undergraduate students enrolled in the RISE program, with a mean age of 21.5 years (SD = 1.96). The participants were largely ethnic Chinese (90%), female (60%), of Buddhist faith (40%), and single (100%). The demographic information of the students is detailed in Table 2. One participant dropped out and did not complete the program. Nine participants completed the post-intervention survey. Students’ narratives are presented in Table 3.
Table 2.
Sociodemographic and academic characteristics of the participants.
| Characteristics | Number of Students | Percentage | Undergraduate Students # | Proportions | |
|---|---|---|---|---|---|
| Age Mean = 21.5, SD = 1.96, Range = 19–24 | |||||
| Gender | Male | 4 | 40% | 15,528 | 49.6% |
| Female | 6 | 60% | 15,744 | 50.4% | |
| Ethnicity | Chinese | 9 | 90% | 74.1% | |
| Others | 1 | 10% | 3.2% | ||
| Religion | Buddhism | 4 | 40% | 33.3% | |
| Christianity | 2 | 20% | 18.3% | ||
| Free thinker | 2 | 20% | 17% | ||
| Islam | 1 | 10% | 14.7% | ||
| Taoism | 1 | 10% | 10.9% | ||
| Average household income | <SGD2500 per person | 3 | 30% | 694,447 * | 50.6% |
| ≥SGD2500 per person | 7 | 70% | 678,113 * | 49.4% | |
| Type of residence | <5-room HDB apartments | 4 | 40% | 765,479 * | 55.8% |
| ≥5-room HDB apartments | 6 | 60% | 530,767 * | 38.7% | |
| Financial aid | Yes | 3 | 30% | 55% | |
| No | 7 | 70% | 45% | ||
| Seniority | Year 1 | 3 | 30% | 8104 | 25.4% |
| Year 2 | 1 | 10% | 7276 | 23.4% | |
| Year 3 | 3 | 30% | 7557 | 25.1% | |
| Year 4 | 3 | 30% | 8230 | 26.1% | |
| Faculty | Arts | 2 | 20% | 3295 | 10.5% |
| Business | 2 | 20% | 4794 | 15.3% | |
| Science | 3 | 30% | 4584 | 14.7% | |
| Engineering | 3 | 30% | 4887 | 15.6% | |
| Nursing | 2 | 20% | 1120 | 3.6% | |
|
Type of
Program |
Single degree program | 7 | 70% | ||
| Non-single degree program | 3 | 30% | |||
| RISE duration 1 | Completion rate (%) | 72 | 24.82 | ||
| Total engagement (min) | 182.66 | 92.85 | |||
#: Data retrieved from 2021 NUS enrolment statistics. *: Data retrieved from 2020 Singapore’s census of population. 1: Mean and standard deviation.
Table 3.
Students’ acceptance, perceived appropriateness and demand, and actual use of the RISE program.
| Categories | Sub-Categories | Quotations |
|---|---|---|
| Acceptance | Overall experience |
|
| Perceived appropriateness | Content |
|
| Take home tasks |
|
|
| Duration |
|
|
| Proposed improvements |
|
|
| Implementation | Facilitators |
|
| Barriers |
|
3.2. Perceived Appropriateness
Regarding the content, all participants found the materials appropriate and applicable to their concerns. In addition, participants appreciated how the take-home tasks were able to consolidate and support their learning. The use of reflective practices was not well-received, and students proposed alternative forms of take-home tasks to stimulate their interest. Regarding the training duration, all participants felt that the duration of six weeks was suitable to deliver the program, although the duration of the training could be shortened. When asked to propose other forms of resilience-enhancing strategies, participants alluded to the importance of skills that fostered their academic coping. Finally, participants also emphasized the need for information relating to outlets for seeking professional support.
3.3. Implementation
The majority of the students verbalized that the training had positive impacts on their resilience, social support, and learning outcomes. In particular, students appreciated how they transferred these skills to their everyday lives. However, the RISE program started during the middle of the academic semester. Students proposed that the training could be initiated at the start of the semester as the current time frame was challenging because of numerous competing academic priorities.
3.4. Demand and Actual Use
The majority of the students were able to complete the videos (n = 8), quizzes (n = 9), and take-home tasks (n = 7) and attended at least two of the three virtual face-to-face discussions (n = 8). This led to a cumulative completion rate of 72% (Range: 60–100%, SD = 28.42), and their total engagement with the RISE program was 182.66 min (Range: 121–360 min, SD = 92.85). Consistent with the students’ narratives and academic timetables, their engagement with the materials was reduced by weeks 5 and 6 of the RISE program, which were also the final two weeks of the academic semester, wherein their workload was heavier.
3.5. Limited Efficacy Testing
The Wilcoxon signed-ranked test was used to compare the differences between the baseline and post-intervention (Table 4). The findings suggested that the RISE program was effective in improving students’ resilience (Z = −2.37, p = 0.01) (Figure 3), with a medium effect size (d = 0.79, 95% CI: −0.15 to 1.74). No significant differences were found for social support (Z = −1.67, p = 0.09) and its subscales (Figure S1). Regarding learning (Figure S2), significant differences were noted for meta-cognitive self-regulation (Z = −2.43, p = 0.01), with a very large effect (d = 1.31, 95% CI: 0.30–2.33). No changes were observed for the other MSLQ subscales.
Table 4.
Effects of the RISE program on the students’ resilience, social support, and learning scores.
| Outcome | Pretest (n = 10) | Post-Intervention (n = 9) | Z | p-Value # | Effect Size (95% CI) 1 | p-Value 2 | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Median | IQR | Range | Mean (SD) | Median | IQR | Range | Mean SD) | |||||
| Resilience | ||||||||||||
| CD-RISC | 64.5 | 60, 73 | 41–83 | 65.4 (11.62) | 76 | 71, 80 | 48–98 | 75.78 (13.4) | −2.37 | 0.02 * | 0.79 (−0.15, 1.74) | 0.10 |
| Social support | ||||||||||||
| MPSS | 5.74 | 5, 5.92 | 2.17–7 | 5.24 (1.31) | 6.17 | 5.67, 6.5 | 1.92–7 | 5.72 (1.51) | −1.67 | 0.09 | 0.33 (−0.58, 1.23) | 0.48 |
| MSPSS-FA | 5.13 | 4.75, 5.25 | 2.5–7 | 4.95 (1.21) | 6 | 5, 6.5 | 2–7 | 5.56 (1.55) | −1.54 | 0.14 | 0.42 (−0.49, 1.34) | 0.36 |
| MSPSS-FR | 5.63 | 4.25, 7 | 3–7 | 5.33 (1.58) | 6 | 5.75, 7 | 2.75–7 | 5.86 (1.35) | −0.81 | 0.43 | 0.34 (−0.57, 1.25) | 0.46 |
| MSPSS-SO | 6 | 5, 6 | 1–7 | 5.45 (1.71) | 6 | 6, 7 | 1–7 | 5.75 (1.85) | −1.98 | 0.13 | 0.16 (−0.74, 1.06) | 0.73 |
| Motivated strategies for learning | ||||||||||||
| MSLQ-CLB | 5.13 | 4.5, 5.75 | 4–6.25 | 5.2 (0.73) | 5.75 | 5.5, 6.25 | 4–7 | 5.81 (0.87) | −1.37 | 0.18 | 0.73 (−0.21, 1.67) | 0.13 |
| MSLQ-SELP | 5.19 | 4.13, 5.5 | 2.7–6.6 | 5.03 (1.18) | 5.75 | 5.13, 6.25 | 3.88–7 | 5.58 (1.02) | −1.68 | 0.11 | 0.47 (−0.44, 1.39) | 0.31 |
| MSLQ-TA | 4.5 | 4.2, 5 | 2.4–6.6 | 4.36 (1.24) | 4.4 | 2.2, 5.2 | 1.2–6.2 | 3.91 (1.74) | −1.36 | 0.20 | −0.29 (−1.19. 0.62) | 0.53 |
| MSLQ-MCSR | 4.5 | 4, 5.08 | 3.3–5.7 | 4.46 (0.76) | 5.42 | 4.92, 5.67 | 4.8–6.5 | 5.40 (0.59) | 2.43 | 0.01 * | 1.31 (0.30, 2.33) | 0.01 * |
| MSLQ-TSE | 5.06 | 4.63, 5.75 | 4.4–6.4 | 5.16 (0.67) | 5.63 | 5.38, 5.75 | 5.4–6 | 5.63 (0.23) | −1.49 | 0.14 | 0.88 (−0.08, 1.83) | 0.07 |
| MSLQ-ER | 5.13 | 4.75, 5.75 | 4.5–6.3 | 5.25 (0.61) | 5.5 | 5, 5.5 | 4.8–6.3 | 5.36 (0.44) | 0.18 | 0.93 | 0.20 (−0.73, 1.12) | 0.68 |
#: Wilcoxon signed-rank test. 1: Difference between the mean and standard deviation scores. 2: Effect size. *: p < 0.05. CD-RISC: Connor-Davidson Resilience Scale; MSPSS: Multidimensional Scale of Perceived Social Support; FA: Family; FR: Friends; SO: Significant Others; MSLQ: Motivated Strategies for Learning Questionnaire; CL: Control of Learning Beliefs; SELP: Self-efficacy of Learning and Performance; TA: Test Anxiety; MCSR: Meta-cognitive Self-Regulation; TSE: Time and Study Environment; ER: Effort Regulation.
Figure 3.
Box plot of resilience scores at the baseline and post-intervention.
3.6. Integration of Qualitative and Quantitative Data
The mixed data analysis [42] comprised qualitative and quantitative data (Figure S3). The qualitative and quantitative data were displayed, reduced, and transformed. Both sets of data were then brought together and compared. The main findings of the mixed-methods analysis concluded that students found the RISE program acceptable and appropriate and that it improved their resilience and meta-cognitive self-regulation. However, modifications were needed with regards to the initiation and contents of the RISE program.
4. Discussion
This mixed-methods feasibility study evaluated the theory-driven, evidence-based, and learner-centered digital resilience training program among undergraduate students. The qualitative results showed that students found the intervention acceptable and appropriate and that they were able to engage with the RISE program. The quantitative results found that students’ resilience and meta-cognitive self-regulation scores improved after the RISE program. In addition, the students’ completion rate was 72%. This figure was comparable to other reported resilience programs whose completion rates ranged from 55% [43] to 78% [44].
Although evaluating the efficacy of an intervention is not the goal of a feasibility study [23], this study found that resilience and meta-cognitive self-regulation scores improved significantly at the post-test. This study found that students’ resilience improved following the RISE program. This was similarly reported in two meta-analyses of resilience programs [16,17]. The positive findings could be attributed to several reasons.
One, the contents of the program were developed based on theories [25,26] where personal, relational, and environmental protective factors of resilience were introduced to improve students’ resilience. Based on the resilience [25] and stress-coping [26] theories, these factors may work in synergy to promote one’s ability to overcome adversities and thus improve their resilience and self-regulation. For instance, the RISE program provided skills that promoted students’ positivity and gratitude, and this was found to improve students’ learning attitudes by allowing them to see the value in and become motivated to exert effort towards learning [45,46]. By becoming motivated toward learning, it is postulated to have a buffering effect against anxiety [47,48].
Two, as part of the developmental process, a qualitative study ensured that the contents and features of the RISE program were cocreated with the target population [28]. With regards to the content, the use of a cocreation approach can ensure the contextual relevance of the intervention and thus ensure that the intervention is relevant to participants’ priorities and needs; this, in turn, improves the outcomes [49,50]. This finding was also supported in two studies evaluating resilience programs [51,52], where the contextual relevance of the resilience programs supported participants’ learning.
Three, with regards to the features, the RISE program was hosted on the university’s learning platform. With reference to the theory of online learning [51], placing a digital program in a platform that is familiar to the target audience can facilitate their participation. In addition, the specific features of the RISE program were developed based on a cocreation process [28], wherein the design features were tailored to students’ preferences, thus enabling their sustained participation.
Four, the RISE program was largely designed as a self-paced program, allowing learners to have flexibility and take ownership of their learning, which, in turn, created a learner-centered environment [53,54]. In addition, the combination of take-home tasks, quizzes, and virtual face-to-face or forum discussions were opportunities to transfer the theoretical knowledge from the classroom to a practical setting. Earlier works concurred with our findings and reported how students value blended learning, as it provides a platform for the application and discussion of newly acquired knowledge [22,53]. Therefore, future digital programs should not merely consist of lectures and discussions but a combination of interactive learning tools, such as practical exercises and quizzes, to engage students.
Nevertheless, students provided several improvements to further promote their engagement in the RISE program. With regards to the take-home tasks, students proposed to have more creative exercises. For instance, reflective practices may be embedded in other forms of activities as opposed to the traditional narrative form. Given that reflective practices are used in discussions [55] and in the form of photos [56], these activities can be considered as a form of reflective practice in future trials.
The findings from our qualitative study [28] proposed a resilience training program be held during the semester and right before major assessments. When enacted in this feasibility study, hosting the RISE program right before major assessments was found to be a challenge for students, especially for those with conflicting priorities. With more evidence showing that the timing of interventions affects learning outcomes [57], alternative initiation time points must be considered for future resilience trials.
4.1. Strengths and Limitations
This study has several strengths and limitations. First, this study adopted a theoretically driven, evidence-based, and learner-centered approach to develop the RISE program. Second, the adaptation of a mixed-methods design allowed a deeper exploration during the evaluation phase of the RISE program, which may not be achievable in a single design. Third, the findings from this feasibility study point toward the development of a larger-scale RCT to evaluate the effectiveness of the RISE program on improving students’ resilience, social support, and learning outcomes.
However, this study was limited by the use of a single-group pretest, post-test design, with the interpretation of its findings limited to the acceptability, appropriateness, and implementation of the RISE program. In addition, the study was only conducted in one university in Singapore. Considering the differences across institutions, the RISE program should be evaluated across multiple IHLs. Further, this study recruited a modest sample of 10 students as it sought to test the feasibility of the RISE program. Further work is needed to confirm the effects of the RISE program among undergraduate students. Finally, due to the COVID-19 pandemic, the use of open-ended questions to collect qualitative data through email correspondences may have limited participants’ responses. The use of in-depth virtual interviews should be considered in future trials to gain a deeper understanding of students’ experiences.
4.2. Implications for Future Research and Practice
This feasibility study lends its findings to the development of future resilience programs. First, larger-scale studies using RCT design are needed to confirm the effects of the RISE program in improving students’ resilience, social support, and learning outcomes. While resilience programs have been found to improve students’ resilience [16], there are limited studies evaluating the effectiveness of resilience training on social support and learning outcomes. The findings from this feasibility study present potential merits that warrant deeper exploration and evaluation.
Second, this study found that the students experienced competing priorities when the RISE program was delivered during the semester. Considering that students prefer to receive resilience training during the academic semester, future programs may be incorporated as part of the formal university curriculum. This may provide students with protected time to engage with the RISE program.
Third, the current RISE program adopts a blended learning design that may be challenging and resource-intensive. Future RCTs may consider comparing the effects of a blended and an asynchronous online resilience program. This has dual benefits, as it may reduce students’ workload and be less expensive to implement.
5. Conclusions
This study examined the feasibility of a digital RISE program. The findings show that students found the RISE program acceptable and appropriate and that they were able to engage with the materials. The limited efficacy testing also show that the digital RISE program can improve students’ resilience (p < 0.05) and meta-cognitive self-regulation (p < 0.05). However, due to the small sample size, it will be challenging to conclude the positive effects found in this study. Instead, this feasibility study paves the way for future studies using RCT designs to provide further evaluations of the RISE program on resilience, social support, and learning outcomes.
Supplementary Materials
The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/ijerph191912729/s1, Table S1: Results of content validation of RISE program among five experts. Table S2: Open-ended questions. Figure S1: Box plots of social support and subscale scores. Figure S2: Box plot of motivated strategies for learning subscale scores. Figure S3: Evaluation of RISE program using a mixed data analysis.
Author Contributions
Conceptualization, W.H.D.A. and Y.L. methodology, W.H.D.A., S.S. and Y.L.; formal analysis, W.H.D.A., H.S.J.C. and Y.L.; investigation, W.H.D.A. and Z.J.Z.; resources, W.H.D.A., L.B.I.S. and E.C.-W.C.; data curation, W.H.D.A., W.H.D.N. and Z.J.Z.; writing—original draft preparation, W.H.D.A. and Y.L.; writing—review and editing, W.H.D.A., S.S., Z.J.Z., W.H.D.N., E.C.-W.C., L.B.I.S., H.S.J.C. and Y.L.; visualization, W.H.D.A. and Z.J.Z.; supervision, Y.L. and S.S.; project administration, W.H.D.A. and Z.J.Z.; funding acquisition, W.H.D.A., S.S., H.S.J.C. and Y.L. All authors have read and agreed to the published version of the manuscript.
Institutional Review Board Statement
The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of National University of Singapore (NUS-IRB-2021-594 on 29 September 2021).
Informed Consent Statement
Informed consent was obtained from all subjects involved in the study.
Data Availability Statement
The data presented in this study are available on request from the corresponding author. The data are not publicly available due to ethical reasons.
Conflicts of Interest
The authors declare no conflict of interest.
Funding Statement
This research was funded by the National University of Singapore Teaching Enhancement Grant [TEG 2021] and the Alice Lee Centre for Nursing Studies Doctoral Research Fund.
Footnotes
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Luthar S.S., Cicchetti D., Becker B. The construct of resilience: A critical evaluation and guidelines for future work. Child Dev. 2000;71:543–562. doi: 10.1111/1467-8624.00164. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Van Breda A.D. Resilience Theory: A Literature Review. South African Military Health Service, Military Psychological Institute, Social Work Research & Development; Pretoria, South Africa: 2001. [Google Scholar]
- 3.Richardson G.E. The metatheory of resilience and resiliency. J. Clin. Psychol. 2002;58:307–321. doi: 10.1002/jclp.10020. [DOI] [PubMed] [Google Scholar]
- 4.Van Breda A.D. A critical review of resilience theory and its relevance for social work. Soc. Work. 2018;54:1–18. doi: 10.15270/54-1-611. [DOI] [Google Scholar]
- 5.Luthar S.S., Barkin S.H. Are affluent youth truly “at risk”? Vulnerability and resilience across three diverse samples. Dev. Psychopathol. 2021;24:429–449. doi: 10.1017/S0954579412000089. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ungar M. Resilience across cultures. Br. J. Soc. Work. 2008;38:218–235. doi: 10.1093/bjsw/bcl343. [DOI] [Google Scholar]
- 7.Chua J.H., Cheng C.K.T., Cheng L.J., Ang W.H.D., Lau Y. Global prevalence of resilience in higher education students: A systematic review, meta-analysis and meta-regression. Curr. Psychol. 2022:1–19. doi: 10.1007/s12144-022-03366-7. [DOI] [PubMed] [Google Scholar]
- 8.Sosu E.M., Pheunpha P. Trajectory of university dropout: Investigating the cumulative effect of academic vulnerability and proximity to family support. Front. Educ. 2019;4:6. doi: 10.3389/feduc.2019.00006. [DOI] [Google Scholar]
- 9.Kermott C.A., Johnson R.E., Sood R., Jenkins S.M., Sood A. Is higher resilience predictive of lower stress and better mental health among corporate executives? PLoS ONE. 2019;14:e0218092. doi: 10.1371/journal.pone.0218092. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Flinn J.T., Miller A., Pyatka N., Brewer J., Schneider T., Cao C.G. The effect of stress on learning in surgical skill acquisition. Med. Teach. 2016;38:897–903. doi: 10.3109/0142159X.2015.1114597. [DOI] [PubMed] [Google Scholar]
- 11.Pascoe M.C., Hetrick S.E., Parker A.G. The impact of stress on students in secondary school and higher education. Int. J. Adolesc. Youth. 2020;25:104–112. doi: 10.1080/02673843.2019.1596823. [DOI] [Google Scholar]
- 12.Frese C., Wolff D., Saure D., Staehle H., Schulte A. Psychosocial impact, perceived stress and learning effect in undergraduate dental students during transition from pre-clinical to clinical education. Eur. J. Dent. Educ. 2018;22:e555–e563. doi: 10.1111/eje.12352. [DOI] [PubMed] [Google Scholar]
- 13.Trammell J.P., Clore G.L. Does stress enhance or impair memory consolidation? Cogn. Emot. 2014;28:361–374. doi: 10.1080/02699931.2013.822346. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Grubic N., Badovinac S., Johri A.M. Student mental health in the midst of the COVID-19 pandemic: A call for further research and immediate solutions. Int. J. Soc. Psychiatry. 2020;66:517–518. doi: 10.1177/0020764020925108. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Wu M., Xu W., Yao Y., Zhang L., Guo L., Fan J., Chen J. Mental health status of students’ parents during COVID-19 pandemic and its influence factors. Gen. Psychiatry. 2020;33:e100250. doi: 10.1136/gpsych-2020-100250. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Ang W.H., Lau S.T., Cheng L.J., Chew H.S., Tan J.H., Shorey S., Lau Y. Effectiveness of resilience interventions for higher education students: A meta-analysis and metaregression. J. Educ. Psychol. 2022;114:1670–1694. doi: 10.1037/edu0000719. [DOI] [Google Scholar]
- 17.Ang W.H.D., Chew H.S.J., Dong J., Yi H., Mahendren R., Lau Y. Digital training for building resilience: Systematic review, meta-analysis, and meta-regression. Stress Health. 2022 doi: 10.1002/smi.3154. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Mulenga E.M., Marbán J.M. Is COVID-19 the gateway for digital learning in mathematics education? Contemp. Educ. Technol. 2020;12:ep269. doi: 10.30935/cedtech/7949. [DOI] [Google Scholar]
- 19.Adedoyin O.B., Soykan E. COVID-19 pandemic and online learning: The challenges and opportunities. Interact. Learn. Environ. 2020:1–13. doi: 10.1080/10494820.2020.1813180. [DOI] [Google Scholar]
- 20.O’Doherty D., Dromey M., Lougheed J., Hannigan A., Last J., McGrath D. Barriers and solutions to online learning in medical education–an integrative review. BMC Med. Educ. 2018;18:1–11. doi: 10.1186/s12909-018-1240-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Tseng S.F., Tsao Y.W., Yu L.C., Chan C.L., Lai K.R. Who will pass? Analyzing learner behaviors in MOOCs. Res. Pract. Technol. Enhanc. Learn. 2016;11:1–11. doi: 10.1186/s41039-016-0033-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Namyssova G., Tussupbekova G., Helmer J., Malone K., Mir A., Jonbekova D. Challenges and benefits of blended learning in higher education. Int. J. Technol. Educ. 2019;2:22–31. [Google Scholar]
- 23.Bowen D.J., Kreuter M., Spring B., Cofta-Woerpel L., Linnan L., Weiner D., Bakken S., Kaplan C.P., Squiers L., Fabrizio C., et al. How we design feasibility studies. Am. J. Prev. Med. 2009;36:452–457. doi: 10.1016/j.amepre.2009.02.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Craig P., Dieppe P., Macintyre S., Michie S., Nazareth I., Petticrew M. Developing and evaluating complex interventions: The new Medical Research Council guidance. BMJ. 2008;337:a1655. doi: 10.1136/bmj.a1655. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Szanton S.L., Gill J.M. Facilitating resilience using a society-to-cells framework: A theory of nursing essentials applied to research and practice. Adv. Nurs. Sci. 2010;33:329–343. doi: 10.1097/ANS.0b013e3181fb2ea2. [DOI] [PubMed] [Google Scholar]
- 26.Lazarus R.S., Folkman S. Stress, Appraisal, and Coping. Springer Publishing Company; New York, NY, USA: 1984. [Google Scholar]
- 27.Ang W.H.D., Shorey S., Hoo M.X.Y., Chew H.S.J., Lau Y. The role of resilience in higher education: A meta-ethnographic analysis of students’ experiences. J. Prof. Nurs. 2021;37:1092–1109. doi: 10.1016/j.profnurs.2021.08.010. [DOI] [PubMed] [Google Scholar]
- 28.Ang W.H.D., Shorey S., Lopez V., Chew H.S.J., Lau Y. Generation Z undergraduate students’ resilience during the COVID-19 pandemic: A qualitative study. Curr. Psychol. 2021:1–15. doi: 10.1007/s12144-021-01830-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Singh H. Challenges and Opportunities for the Global Implementation of E-Learning Frameworks. IGI Global; Pennsylvania, PA, USA: 2021. Building effective blended learning programs; pp. 15–23. [DOI] [Google Scholar]
- 30.Pealer L.N., Dorman S.M. Evaluating health-related Web sites. J. Sch. Health. 1997;67:232–235. doi: 10.1111/j.1746-1561.1997.tb06311.x. [DOI] [PubMed] [Google Scholar]
- 31.Connor K.M., Davidson J.R. Development of a new resilience scale: The Connor-Davidson resilience scale (CD-RISC) Depress. Anxiety. 2003;18:76–82. doi: 10.1002/da.10113. [DOI] [PubMed] [Google Scholar]
- 32.Windle G., Bennett K.M., Noyes J. A methodological review of resilience measurement scales. Health Qual. Life Outcomes. 2011;9:1–18. doi: 10.1186/1477-7525-9-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Zimet G.D., Dahlem N.W., Zimet S.G., Farley G.K. The multidimensional scale of perceived social support. J. Personal. Assess. 1988;52:30–41. doi: 10.1207/s15327752jpa5201_2. [DOI] [Google Scholar]
- 34.Akhtar A., Rahman A., Husain M., Chaudhry I.B., Duddu V., Husain N. Multidimensional scale of perceived social support: Psychometric properties in a South Asian population. J. Obstet. Gynaecol. Res. 2010;36:845–851. doi: 10.1111/j.1447-0756.2010.01204.x. [DOI] [PubMed] [Google Scholar]
- 35.Zimet G.D., Powell S.S., Farley G.K., Werkman S., Berkoff K.A. Psychometric characteristics of the multidimensional scale of perceived social support. J. Personal. Assess. 1990;55:610–617. doi: 10.1080/00223891.1990.9674095. [DOI] [PubMed] [Google Scholar]
- 36.Pintrich P.R. A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ) University of Michigan; Ann Arbor, MI, USA: 1991. [Google Scholar]
- 37.Pintrich P.R., Smith D.A.F., Garcia T., McKeachie W.J. Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ) Educ. Psychol. Meas. 1993;53:801–813. doi: 10.1177/0013164493053003024. [DOI] [Google Scholar]
- 38.StataCorp LLC Stata data analysis and statistical Software. Spec. Ed. Release. 2021;10:733. [Google Scholar]
- 39.Sullivan G.M., Feinn R. Using effect size—Or why the P value is not enough. J. Grad. Med. Educ. 2012;4:279–282. doi: 10.4300/JGME-D-12-00156.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Mayring P. Qualitative content analysis. A Companion Qual. Res. 2004;1:159–176. [Google Scholar]
- 41.McHugh M.L. Interrater reliability: The kappa statistic. Biochem. Med. 2012;22:276–282. doi: 10.11613/BM.2012.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Onwuegbuzie A.J., Teddlie C. A framework for analyzing data in mixed methods research. Handb. Mix. Methods Soc. Behav. Res. 2003;2:397–430. [Google Scholar]
- 43.Joyce S., Shand F., A Bryant R., Lal T.J., Harvey S.B. Mindfulness-based resilience training in the workplace: Pilot study of the internet-based Resilience@ Work (RAW) mindfulness program. J. Med. Internet Res. 2018;20:e10326. doi: 10.2196/10326. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Chesak S.S., Khalsa T.K., Bhagra A., Jenkins S.M., Bauer B.A., Sood A. Stress Management and Resiliency Training for public school teachers and staff: A novel intervention to enhance resilience and positively impact student interactions. Complement. Ther. Clin. Pract. 2019;37:32–38. doi: 10.1016/j.ctcp.2019.08.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Wilson J.T. Brightening the mind: The impact of practicing gratitude on focus and resilience in learning. J. Scholarsh. Teach. Learn. 2016;16:1–3. doi: 10.14434/josotl.v16i4.19998. [DOI] [Google Scholar]
- 46.Howells K. Gratitude in Education. Sense Publishers; Rotterdam, The Netherlands: 2012. [Google Scholar]
- 47.Basco L.M., Han S.H. Self-esteem, motivation, and anxiety of Korean university students. J. Lang. Teach. Res. 2016;7:1069. doi: 10.17507/jltr.0706.02. [DOI] [Google Scholar]
- 48.Turel Y.K., Sanal S.O. The effects of an ARCS based e-book on student’s achievement, motivation and anxiety. Comput. Educ. 2018;127:130–140. doi: 10.1016/j.compedu.2018.08.006. [DOI] [Google Scholar]
- 49.Halvorsrud K., Kucharska J., Adlington K., Rüdell K., Brown Hajdukova E., Nazroo J., Haarmans M., Rhodes J., Bhui K. Identifying evidence of effectiveness in the co-creation of research: A systematic review and meta-analysis of the international healthcare literature. J. Public Health. 2021;43:197–208. doi: 10.1093/pubmed/fdz126. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Palmer V.J., Weavell W., Callander R., Piper D., Richard L., Maher L., Boyd H., Herrman H., Furler J., Gunn J., et al. The participatory zeitgeist: An explanatory theoretical model of change in an era of coproduction and codesign in healthcare improvement. Med. Hum. 2019;45:247–257. doi: 10.1136/medhum-2017-011398. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51.Corlett J., McConnachie T. Delivering resilience training to pre-registration student nurses in partnership with a reservist military organisation: A qualitative study. Nurse Educ. Today. 2021;97:104730. doi: 10.1016/j.nedt.2020.104730. [DOI] [PubMed] [Google Scholar]
- 52.Giovannetti A.M., Quintas R., Tramacere I., Giordano A., Confalonieri P., Messmer Uccelli M., Solari A., Pakenham K.I. A resilience group training program for people with multiple sclerosis: Results of a pilot single-blind randomized controlled trial and nested qualitative study. PLoS ONE. 2020;15:e0231380. doi: 10.1371/journal.pone.0231380. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Anderson T. Towards a theory of online learning. In: Anderson T., editor. The Theory and Practice of Online Learning. Athabasca University Press; Edmonton, AB, Canada: 2004. pp. 45–74. [Google Scholar]
- 54.Nortvig A.-M., Petersen A.K., Balle S.H. A Literature Review of the Factors Influencing E-Learning and Blended Learning in Relation to Learning Outcome, Student Satisfaction and Engagement. Electron. J. E Learn. 2018;16:46–55. [Google Scholar]
- 55.Pretorius L., Ford A. Reflection for Learning: Teaching Reflective Practice at the Beginning of University Study. Int. J. Teach. Learn. High. Educ. 2016;28:241–253. [Google Scholar]
- 56.Hahl K. Student teachers’ experiences of using photos in teacher reflection. Reflective Pract. 2021;22:115–127. doi: 10.1080/14623943.2020.1854212. [DOI] [Google Scholar]
- 57.Nguyen Q., Huptych M., Rienties B. Linking students’ timing of engagement to learning design and academic performance; Proceedings of the 8th International Conference on Learning Analytics and Knowledge; Sydney, Australia. 7–9 March 2018. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data presented in this study are available on request from the corresponding author. The data are not publicly available due to ethical reasons.