A digital literacy model to narrow the digital literacy skills gap

Abstract

The concept of digital literacy is increasingly prevalent in the 21st century. Growing demands from the work sector for individuals to be digitally literate has prompted targeted interventions and innovations from the education sector to instil digital skills into the future workforce. However, despite efforts the digital skills gap remains visible globally. This paper explores the prominent educational frameworks and models, their advantages and limitations in 21st-century learning and teaching. Furthermore, a new innovative digital literacy model has been proposed to be integrated into the existing and future education frameworks and models to assist educationists in narrowing the digital skills gaps and preparing graduates for the work sector. The digital literacy model consists of two components: (1) the digital literacy framework- South Pacific Digital Literacy Framework (SPDLF) and (2) the digital literacy tool. The SPDLF reflects six major literacies identified for the 21st-century while the digital literacy tool--digilitFJ consists of a digital literacy measuring scale and an online intervention program. The exploratory factor analysis showed that the SPDLF was valid. On the other hand, heuristics, student attitude, and satisfaction and effectiveness of the digital literacy tool were also evaluated from the student's perspective to reflect its usefulness. The survey results also showed a positive attitude and perception of the use of the tool. Additionally, Cohen's d value showed that the digital literacy tool was effective. Therefore, if the tool is implemented and adopted, it can narrow the existing digital skills gap in the South Pacific.

1. Introduction

The intensification of technology and its use in digitalization stimulated the need for digital education. The reliance on digital technology is rapidly increasing and much consideration needs to be given to how individuals use digital technology, how individuals interact online and the skills individuals' possess to carry out the tasks associated with digitalization. Additionally, the work sector demand for labour qualifications has transformed with digital skills becoming a strong requirement for the sustainability of businesses, the livelihood of workers, and the growth of economies. As such, the education institutes play the facilitator role in imparting appropriate digital skills to the future workforce through innovative learning trajectories.

Over the years, the education sector has implemented various strategies to improve the learning as well as digital skills of their learners and prepare them for technology-driven workplaces. Different 21st-century educational frameworks and models have been used to idealize the vision of the current and future workforce. Some prominent 21st-century education frameworks and models include the Organisation for Economic Co-operation and Development (OECD) learning framework, the E³ learning model, the Partnership for 21st Century Skills (P21) framework, and many others, which are generally termed as 21st-century learning models or 21st-century learning frameworks [1]. These frameworks and models articulate the relevant competencies and act like "signposts" guiding 21st⁻ century learning. Furthermore, the Technological Pedagogical Content Knowledge (TPACK) and Substitution, Augmentation, Modification, and Redefinition (SAMR) frameworks have played a significant role in guiding educators to successfully integrate technology in the learning process [2].

However, recent research has highlighted that such and other frameworks and models have mostly been seen to be ineffective since high attrition rates at higher education still persist [[3], [4], [5], [6], [7], [8]]. Learners lack relevant digital skills and knowledge thus face difficulties in coping with technology-enabled learning. Therefore, they are either unable to progress in their studies or drop out of the learning system [9,10]. The work of [9,10] state that digital literacy is the solution to curb high attrition rates in online learning and prepare digitally literate graduates for workplaces. At workplaces, individuals with digital skills tend to perform better and are more productive than those who lack digital skills [11,12]. Organizations are also aware of the gap between existing and needed digital competencies [13]. Many employers are organizing digital workshops and training to ensure that their workforce has relevant digital competencies needed for the digitalized future [13,14]. For example, (1) in Russia and Kazakhstan, businesses are developing business models which reflect the necessary skills and competencies of industry 4.0, (2) Germany organized national programs for its digital economy, and (3) Singapore motioned the smart nation program to take into account the skills and competencies of the digital economy [15]. Additionally, the Covid-19 pandemic stimulated the crucial need for individuals, organizations, societies and economies to address the digital skills gap for survival. Efforts such as apprentices, government fast-track initiatives to improve digital literacy, employing temporary digitally skilled workers, or remotely-based freelancers have failed [13]. As such a holistic and targeted approach needs to be taken to address the issue of the existing digital skills gap.

Motivated by the inability to curb and curtail gaps in digital literacy, the current study reviews the prominent education model which have been utilized when designing the above-mentioned efforts from the private and public sectors and then proposes a validated digital literacy model that can be used by education institutes or any organization to prepare individuals for a technology-enabled environment. For education institutes, the proposed digital literacy model can be integrated with the existing and future educational frameworks and models. At the same time, other organizations can assimilate the proposed digital literacy model into or with the existing and future frameworks they develop. The validated digital literacy model consists of two major components; (1) a validated digital literacy framework named South Pacific Digital Literacy Framework (SPDLF), which has six different literacies that are relevant for survival in the 21st-century, and (2)a digital literacy tool named digilitFJ which comprises of a digital literacy measuring scale and a digital literacy intervention program.

The current article articulates and contributes as follows.

• i.Reviews the prominent education frameworks and models, namely; OECD, E3, P21, TPACK and SAMR, which education institutes currently use to facilitate technology-enabled learning. Researchers state that the current educational frameworks and models for technology-enabled learning have flaws that contribute to high attrition rates in online learning and growing digital skills gap [ [4,7,8]]. This research paper presents a comparative analysis of the selected models and reflected the advantages and existing gaps of the popular educational models. The identified gaps are the reasons for high attrition rates. The study recommends a digital literacy model to be assimilated with the currently referred educational models.
• ii.Proposes a validated digital literacy model that can be used by educationists or relevant stakeholders to address the challenges of digital skills gap amongst individuals and the gaps identified that has been identified in the comparative analysis. The existence of digital skills gap has been a challenge for both the facilitators and the students in the technology-enhanced learning environment [5,9,15]. As such, the authors believe that the proposed digital literacy model will resolve the existing digital skills gap amongst individuals and as well the as it is deeply infused with the relevant digital literacy skills critical and essential for 21st-century survival.

Presents a quantitative evaluation of the effectiveness of the proposed digital literacy model through evaluating user satisfaction on the heuristics of the digital literacy tool, user attitude and perception and the effectiveness of the digital literacy tool. This study depicts that the proposed digital literacy model is effective and can be used to narrow the existing digital literacy gap in the Pacific and beyond.

The article is structured as follows: the literature review section consists of a comprehensive review of the most prominent educational frameworks and models in the 21st-century and concludes with the proposed digital literacy model that can be integrated with the current and existing educational frameworks and models. The methodology section describes the digital literacy tool, the intervention program and the method for conducting the study. The results section consists of the response from the students after which the article ends with a conclusion.

2. Literature review

2.1. Digital literacy and educational frameworks and models

The ongoing proliferation of digital technology and services has led individuals to be either digitally included or digitally excluded. Individuals who are not digitally literate have a higher tendency to not be able to handle the various domains of their lives, thus increasing the likelihood of struggling to participate and survive in the digital world and be compatible with digitalization Researchers such as [14, [16], [17], [18], [19], [20], [21], [22], [23]] highlight that the growing use of digital technologies, digital platforms and technology-enabled services arises the need for citizens to possess relevant digital literacy skills to carry out their designated tasks effectively. Digital literacy education has become a worldwide agenda for policymakers, education stakeholders, and researchers [7]. Internationally, governments and policymakers have made digital literacy an asset of the education system. Integrating digital literacy with the education curriculum will enable the education institutes to prepare their future graduates/workforce for technology-enabled workplaces [19]. Research shows that the concept of digital literacy evolved with the invention and growing use of new digital technologies [8,19,20]. Therefore, the educational frameworks and models designed to cater for digital literacy education must reflect appropriate digital literacy skills. The following paragraphs will discuss the evolution of digital literacy.

To begin with, extensive efforts have been made to describe and conceptualize digital literacy skills, a term first defined by "Paul Glister" in 1997. The readers are referred to the work of [14], which provides an in-depth discussion of the evolution of digital literacy definition with time. In more recent times, digital literacy is conceived as a combination of technical-procedural (e.g., handling files and editing visuals), cognitive (e.g., the ability to intuitively decipher or "read" visual messages embedded in graphic user interfaces, working with search engines, evaluating data, sorting out false and biased data, and distinguishing between relevant and irrelevant data) and emotional-social (e.g., ability to communicate online safely, being aware of the legal and ethical guidelines of using digital platforms) skills. Professionals and subject experts developed comprehensive, coherent, and rigid frameworks that guided digital literacy education to improve the understanding of digital literacy and further enable educational institutes and academics to advocate digital literacy skills. Numerous digital literacy frameworks have been published in the literature to cover meaningful skills that 21st-century citizens are required to have to successfully function in a digital-oriented society. The digital literacy skills for the digital literacy frameworks and models were selected by researchers according to the context of the study or the societal needs or the economic needs and background. Given below are popular digital literacy frameworks from 2017 to 2020 designed by.

• i[20]. that comprised of operational skills, technical skills, collaboration skills and awareness skills,
• ii[19]. that comprised of computer literacy, network literacy, information literacy, online reading literacy, web literacy., communication and collaboration literacy and informatics literacy,
• iii[8]. that comprised knowledge assembly, content evaluation, Internet searching and hypertextual navigation.
• iv[14]. which comprised of media literacy, communication literacy, information literacy, visual literacy, technology literacy and computer literacy.

The area of digital literacy became indefinable with the emergence of new digital technology, tools and new literacies. As such, greater clarification and consensus was needed to understand both functional and critical dimensions of its changing nature. While the exisiting digital literacy frameworks and models (as mentioned above) provided the underlying principles of digital literacy, they did not explain the teaching models and learning theories that ultimately formed the basis of the teaching strategies [ [5]] [9]. To fill this gap, different educational models and frameworks emerged from different researchers, educationists, experts and professionals based on their personal experiences, academic grounding, cultural and societal background and curriculum content [ [3]] [5,9,24,25]. A few of these educational models became popular and were used by educationists for technology-enabled learning. However, studies have shown that the issue of digital skills gap is still eminent as educators are still struggling with the attrition rates and the workpace employers trying to continuously upskill the digital skills of their employees due to the changing nature of the work environment. This article intends to review the commonly used educational models to identify the digital skills gaps that still exists in the education institutes and the work environment. The next section provides a critical review of the popular educational models.

2.2. Review of the educational frameworks and models

Experts and relevant stakeholders designed the 21st-century educational frameworks and models to idealize the vision of the 21st-century digitally literate citizens who would contribute towards the economic development of a country [5]. The frameworks and models were designed through the examination of social and cross-cultural skills, intellectual understanding, relatability, relevant competencies required, background, means of education, and economic returns. Numerous frameworks and models were infused into the education system globally, of which the most prominent ones will be explored in this paper.

i. The education models of The Organisation for Economic Co-operation and Development (OECD) are known for their contributions in reshaping global education for decades. The OECD's role in education involves a measure of innovations in education, change in pedagogical practices inside and outside of classrooms and evolution of teaching practices. Afterwhich, the organization makes recommendations and suggestions to policymakers and the education community on ways and means to improve the quality of the education sector. As of July 2020, OECD has 79 education specific indicators highlighted and these details current issues in the education sector [ [1]]. The OECD learning compass 2030 is the most recent learning framework in place that supports the wider goals of education and provides points of orientation towards future education [2].

Furthermore, the learning compass model, as shown in Fig. 1, describes the ability of students to learn independently and responsibly. The components include core foundations, knowledge, skills, attitudes and values, transformative competencies and a cycle of anticipation, action and reflection, which can be used to develop competencies such as financial literacy, global competency and media literacy [24]. The model also envisions the competencies that students must have by 2030 and beyond for economic prosperity and societal well-being. Countries like Australia, Canada, Finland, New Zealand, and Singapore have implemented the OECD learning compass 2030 components into their curriculum to improve their educational pedagogies.

Fig. 1.

The OECD 2030 learning compass. Adapted from Ref. [25].

ii. The E³ learning model (entrepreneurship, employability, education) uses a student-centred learning approach and incorporates projects and games in the existing learning curriculum to prepare the learners for the 21st - century (Basic Education, Republic of South Africa, 2020) Fig. 2 shows the E³ learning model – a cycle model which intends to incorporate the 21st-century competencies in scholars. The model focuses on the project-based learning process for reinforcing real-life knowledge and experience, encourages learners to be critical and independent thinkers and workers, gain confidence and become problem-solvers [26] The mentioned report states that the E³ learning model is a response to the changes of the 4th industrial revolution and intends to meet the goals of the new learning domains which are defined as 21st-century competencies/skills/values/attitudes/ethics, soft skills, life skills, social-emotional skills, intra-personal and inter-personal skills, and global competences. Twenty schools in South Africa used the E³ learning model to develop 21st-century skills in learners through project-based learning and games [27]. The E³ programme, therefore, promotes the development of an entrepreneurial mindset in all subjects, through its constructivist pedagogy and contributes to educating all learners to become Entrepreneurial [28].

Fig. 2.

The E³ learning model. Adapted from Ref. [26].

ii. Partnership for 21st Century Learning (P21) framework is a 21st-century learning model that was developed in collaboration with teachers, education experts, and business leaders to clearly describe, visualize the skills and knowledge that learners need for the 21st century [29]. The framework describes the skills, knowledge and expertise students must master to succeed in work and life; it is a blend of content knowledge, specific skills, expertise and literacies [30]. In the United States, the P21 learning framework was used to educate the scholars on the 21st-century interdisciplinary themes [31]. Fig. 3 shows the P21 framework and to bring out the details, the subjects of each theme has been added [31].

Fig. 3.

The P21 learning framework [31].

P21 advocates for a variety of assessments that promote creativity, critical thinking, communication and collaboration, such as standardized testing and project and problem-based assessments. According to Ref. [31], more than 800 education institutes in the united states are using the P21 framework, and researchers have recommended that the developing countries use this framework to narrow the barriers to teaching and learning [ [30]].

iv. The TPACK and SAMR models were developed for teachers to change their teaching attitudes towards integrating technologies into teaching practices. Since technological innovations were impacting the teaching and learning processes and transferring of knowledge was from teachers to learners, the teachers were also required to have competencies and a good comprehension of digital concepts and theories of teaching [32]. Fig. 4 illustrates the TPACK model (on the left) and SMAR model (on the right).

Fig. 4.

The TPACK and SMAR model. Adapted from Ref. [32].

Literature shows that the TPACK and SAMR models complement each other [3,33]. For teachers to integrate technology into their teaching practices and revise their teaching and learning pedagogies, they incorporate the competencies shown in the TPACK model-technology knowledge, content knowledge and pedagogical knowledge. Further on, the levels in the SAMR model – redefinition, modification, augmentation and substitution combined with the levels of bloom's taxonomy create, evaluate, analyse, apply, understand and remember can be used to critically implement the learning activities for students [32]. Although the SMAR and TPACK models improved teacher knowledge on technology and technology-enabled pedagogies, both models have apparent challenges and issues. Both models provided positive and valuable insights to drive new opportunities for student learning, however, insufficient teacher knowledge, skills, confidence, acceptance and technical support hindered the transformation of this knowledge and skills [3,34,33]. Furthermore [34], states that many teachers lacked practical technical knowledge, therefore, found the TPACK model complicated to use to enhance their courses.

[35] state, "designing and facilitating robust learning in technology-enhanced environments is complex and complicated if one does not possess relevant skills". An important observation from the literature was that those teachers who lacked digital literacy skills faced challenges and issues using the TPACK and SAMR models. For example, the study conducted by Ref. [3] in Russia, reflected that deficient digital skills led to usage and outcome gap for the teacher users. Similar results were obtained for studies conducted by Refs. [35,36] which showed that teachers preferred to improve their digital competencies before they got involved in using educational models to improve their delivery. Other issues included lack of appropriate technology integration training, limited access to instructional technology, lack of technical support, and the theft of instructional technology resources [4, 36, 37].

To sum up, while the uptake of discussed models and frameworks for learning in the 21st century was beneficial and successful, a critical analysis of the literature showed that the models are not without limitations. The 21st-century education models displayed modern and life-long learning approaches to teaching and learning but the digital skills gap of both the teachers and students still exists [18,37]. Globally, digital literacy frameworks and digital literacy programs have also been developed [7,38] and implemented but individuals still lack digital literacy skills. As such experts suggested that digital literacy be made part of the educational curriculum [5,8,9,20]. According to Refs. [5,37] digital literacy improves classroom performance, staff and student readiness to technology-enabled learning and prepares graduates for the technology-enabled work environment. The educational models have components that intend to improve the digital skills of individuals (teachers or students), however, the issue of the digital skills gap was still not resolved [5,9]. The continued presence of the digital skills gap proves that the components in the educational models which intended to improve individuals digital skills were not effective. A more sophisticated and innovative element that would improve digital literacy skills was needed. Hence, this study proposes a digital literacy model that can be incorporated into existing and future educational models. The proposed digital literacy model has been validated and successfully used to evaluate the digital literacy of high school and tertiary students [38]. The proposed model consists of the relevant digital literacy skills that are essential in the 21st-century and the authors of this paper believe that if implemented appropriately, the issue of the existing digital skills gap can be curbed and curtailed.

2.3. Proposed educational model

The digital literacy model that is proposed for this study consists of (1) a digital literacy framework, and (2) a digital literacy tool--digilitFJ. Fig. 5 illustrates the proposed digital literacy model. The digital literacy framework is an umbrella that comprises six other literacies namely, media literacy, information literacy, technology literacy, visual literacy, communication literacy and computer literacy. The framework for digital literacy has been adopted from the work of Covello (2010) [38] and further revised to suit the relevant skills needed nowadays. Furthermore, the definitions of the six literacies and digital literacy were also enhanced in light of the evolution and use of digital technologies. The six literacies are defined as follows (adopted from [ [11,38]].

• 1.Information Literacy: Using digital technology to find, locate, analyse and synthesise resources, evaluating the credibility of these resources appropriate citation techniques, abiding the legal and ethical issues surrounding the use of these resources and formulating research questions in an accurate, effective and efficient manner.
• 2.Computer Literacy: An understanding of how to use computers, digital technologies and their applications for practical use.
• 3.Media Literacy: Having the ability to use digital technologies to access, analyse, evaluate and communicate information in a variety of digital platforms.
• 4.Communication Literacy: Using digital technologies to communicate effectively as individuals and work collaboratively in groups, using publishing technologies, the Internet and Web 2.0 tools and technologies.
• 5.Visual Literacy: Having the ability to use digital technology to ‘read,’ interpret, and understand information presented in pictorial or graphic images communicate this information and convert the information into visual representations.
• 6.Technological Literacy: Having the ability to use digital technology to improve learning, productivity and performance.

Fig. 5.

Proposed digital literacy model.

The authors [38] also associated appropriate skills to each literacy to further enhance the digital literacy framework. The digital literacy tool, which consists of the digital literacy scale and the digital literacy intervention program was then developed using the digital literacy framework. The digital literacy tool consists of a self-evaluating questionnaire that can be used to measure the digital literacy skills of individuals. The digital intervention program, which is a contextualized game-based intervention, was further developed to improve the digital literacy skills of individuals. More details are in the methodology section and the readers are also referred to the work of [11] for a detailed description of the Digital Literacy Intervention Programme (DLIP). The work of [11] Reddy et al. (2021a) shows that remediation interventions are important as these develop and facilitate critical thinking and learning abilities of students and furthermore these skills and knowledge assist the students when they join the workforce. Digital literacy interventions such as Digital literacy programs in Sierra, Internet Literacy Program (ILP), Microsoft online digital literacy courses, Digital literacy assessment developed by Literacy Minnesota, . E & S Online assessment tool provided by the joint effort of OECD [38] have been developed and are available, however, there is still presence of the digital skills gap.

The proposed digital literacy model can be used to narrow the digital skills gap mentioned in the "Review of the educational frameworks and models" section. The model has been designed to include the must-have competencies and skills in the 21st-century. The authors believe that the model will narrow the identified digital literacy gaps amongst the learners, facilitators, workforce and even the commoners in a society. To add on, the digital literacy models of [8,39] had elements of digital literacy missing and the skills that were tested for were limited in scope. These authors had envisioned the changing nature of digital literacy and therefore had recommended that their digital literacy models could be further improved in future. As such, with the digital literacy framework and the digital literacy tool being validated and successfully used to evaluate the digital literacy competencies of individuals in the Fiji Islands, these are recommended to be integrated with the educational models for the 21st-century [11,38]. If implemented appropriately, the digital skills gaps and the challenges faced by facilitators can be narrowed. As mentioned in the section "Review of the educational frameworks and models" facilitators face issues while implementing the educational models for technology-enabled learning due to their lack of digital literacy skills. As shown in Fig. 5, the proposed model is a solution to the existing issue or challenges that the facilitators face. Furthermore, suppose the facilitators improve their technical skills. In that case, they will implement smarter pedagogies for learning in the technology-enabled learning environment [37]. Therefore, student learning will improve and there will be a reduction in the attrition rates at educational institutes, which were increasing due to a lack of technical skills in technology-enabled learning [4,36].

The study proposes the model shown in Fig. 5 be integrated with the existing and future educational model.

Previous work done on the proposed digital literacy model has shown that the model is effective in evaluating the digital literacy skills of individual's [11,38]. The article will further evaluate the user satisfaction on heuristics of the digital literacy tool (DLS and DLIP), the attitude and perception towards the digital literacy tool (DLS and DLIP) and the effectiveness of the tool. Heuristic evaluation is one of the most widely-used methods for evaluating the usability of online systems, which was introduced in 1990 by Nielsen and Molich [40]. Furthermore, the authors state that the usability of a system is evaluated to estimate the extent to which users can use the application to achieve the set goals effectively, efficiently and satisfactorily. The work of [41] adds that the perceived usability of technological systems affects the learning experience and effectiveness of the system. Literature shows that Neilson's heuristics is the most common and effective method to evaluate the usability of a system, therefore, used to evaluate the usability of DLS and DLIP. More to this, student attitude, perception and effectiveness of DLS and DLIP were evaluated.

2.4. Research questions

The article works on the following questions to show effectiveness of the proposed digital literacy model for this study.

• i.Are the users satisfied with the heuristics of the digital literacy scale?
• ii.Are the users satisfied with the heuristics of the digital literacy intervention program?
• iii.What is the attitude and perception of the users to the digital literacy tool?
• iv.Is the digital literacy tool effective?
• v.Is there any correlation between the three factors– the satisfaction on the heuristics of the digital literacy scale and the digital literacy intervention program and the attitude and perception of the students to the digital literacy tool?

3. Methodology

A quantitative survey method was used to evaluate student satisfaction for the study. To evaluate the student satisfaction with the newly designed digital literacy tool, a 5 point Likert scale online questionnaire was administered to the users. A total of 105 users gave their feedback on the digital literacy tool. The ethics approval for the research was given by the University of the South Pacific research committee. The data were analyzed using Microsoft Excel and the mean and standard deviation were calculated. The digital literacy model consists of: (1) the South Pacific digital literacy framework (SPDLF), (2) the digital literacy measuring tool--digilitFJ. The digital literacy tool is further divided into the digital literacy scale (DLS), and the digital literacy intervention program (DLIP). The DLS was developed using the SPDLF, which consists of six literacies. It measures digital literacy competencies of individuals using a validated self-evaluated questionnaire. The DLIP has gamified remediation modules that consist of remediation contents for the six literacies. For further details on digilitFJ, the reader is referred to the prequel papers titled; “Essaying the design, development and validation processes of a new digital literacy scale” and “Contextualized game-based intervention for digital literacy for the Pacific Islands”.

The components of the digital literacy model have been validated and results are presented below within four sequential subpoints.

• i.The digital literacy framework (SPDLF), consists of six literacies as mentioned above (Media literacy, Information Literacy, Visual Literacy, Communication Literacy, Technology Literacy and Computer Litercay). In total there are 60 attributes or self-evaliuated questions, meaning each literacy having its own set of attributes or self-evalaued questions. The framework was validated using an Exploratory Factor Analysis (EFA). The 60 attributes were carefully selected from literature [8,[42], [43], [44], [45], [46]]. The details of the 60 attributes are included in the paper tilted “Essaying the Design, Development and Validation Processes of a New Digital literacy Scale”
• ii.The factor loadings for the sixty attributes ranged from 0.3 and above. Literature shows that variables with factor loadings above 0.5 are good to be used [47], while some researchers state that variables that have factor loadings below 0.5 can also be kept [48]. Furthermore, multiple linear regression results indicated that all the 60 attributes were statistically significant contributors to digital literacy. Thus the authors of the current study decided to keep all the variables in the digital literacy framework.
• iii.The measuring instrument namely or the digital literacy scale is self-evaluating and ranges from "No understanding – Advanced level of understanding". Each response was allocated points as follows: "No understanding – 1", "Low understanding – 2", “Average understanding = 3”, “High level of understanding = 4”, “Advanced level of understanding = 5”. Therefore, the maximum points for each response are 5 and the minimum is 1. In total, the maximum points achieved by anyone can be 300 and a minimum of 60 points. Using the class interval method, which in literature was the most appropriate method for early measuring instruments [39,49], a digital literacy scale was developed. Six levels were derived for the scale and the minimum point is 0 and the maximum is 60. The reader is referred to the work of [11] for additional details. The descriptors of DLS are shown in Table 1.

Table 1.

Description of the digital literacy scale.

Levels	Points	Description
L1	0–10	No Understanding
L2	11–20	Very Low
L3	21–30	Low
L4	31–40	Average
L5	41–50	High
L6	51–60	Very high (Expert)

iv. DLIP was developed to improve the digital literacy skills of individuals. The DLIP has been developed using a game-based intervention design process. It consists of six modules (from SPDLF) which are divided into two parts: (1) learning through theory and (2) testing the knowledge through a quiz. The DLIP is linked to DLS so users can compare their self-evaluated scores with their scores from the quiz in the DLIP. As shown in Fig. 6, DLIP allows the user to log in, review each module's theoretical concepts, and then attempt the game-based quiz. There is an online badge that the user can collect at the end of each module. The theoretical component also has coins to collect – this was done to motivate the user to read the theoretical content. The user can view their scores for each module at anytime they want. Once all the modules are complete, the user can view his/her progress using the dashboard, which shows the self-evaluated and actual score, coins collected and the badge for each module. The final digital literacy level is also shown on the dashboard. Moreover, once all the modules are completed the user can print his/her certificate.

Fig. 6.

Steps involved to complete DLIP.

Table 2 shows the digital literacy levels with the mandatory requirements. If an individual is at L1, this means that he/she must pass 1 module with a minimum score of 5. For L2, an individual must pass 2 modules meaning a minimum score of 5 in each module, and the same formulation applies to the rest of the levels in the DLIP.

Table 2.

DLIP levels and passing requirements.

Levels	Modules Passed	Minimum Total Score
L1	1	5
L2	2	10
L3	3	15
L4	4	20
L5	5	25
L6	6	30

3.1. Measuring instrument

To measure the effectiveness of the digital literacy tool, a unipolar 5 point Likert scale questionnaire ranging from “strongly disagree – strongly agree” was designed. Literature shows that heurisitic evaluations are used to measure the usability of a system. Prior studies conducted on heuristic evaluation have used Neilsen’s 10 usability heuristics [ [[50], [51], [52], [53], [54]]]. Since, Neilsen’s measure has been effective, the authos of this paper have used the same approach to measure the heurisitics/usability of the digital literacy tool [ [[50], [51], [52], [53], [54]]].

The questionnaire was divided into three sections.

• i.Factor 1: Heuristic Evaluation of Digital Literacy Scale (DLS)- further divided into (1) Appearance/Aesthetics: the first impression of the application, (2) Content: easy for users to find and use the application, (3) Navigation: makes getting around the application easy.
• ii.Factor 2: Heuristic Evaluation of Digital Literacy Intervention Program (DLIP)- further divided into (1) Appearance/Aesthetics: the first impression of the application, (2) Content: easy for users to find and use the application, (3) Navigation: makes getting around the application easy.
• iii.Factor 3: Student Attitude and Perception on the Digital Literacy Tool – further divided into student attitude, student perception - perceived usefulness, perceived ease of use.

The three factors will be used to evaluate user satisfaction for the digital literacy tool. The questions were adopted from relevant literature and modified to suit the study [52,55]. The questionnaire was first piloted with 20 students and a Cronbach alpha test was performed. The Cronbach alpha for the test was 0.90. Therefore, the questionnaire was reliable to be used. The content was valid as the students did not face any difficulty in attempting the questions.

3.2. Sample

The online survey was administered to the 1st – year university students in Fiji. Since the participation was voluntary, only 105 students participated. A total of 62.1% of the females and 37.9% of the males participated. Majority of the respondents were from the age group of 17–25 years (77.3%). All the students were enrolled into degree programmes of which 71.5% were in Arts programs whereas 28.5% were in Science programs.

3.3. Data analysis

A total of 105 responses was gathered and the data were analyzed using Microsoft Excel to evaluate the mean and the standard deviation. Student satisfaction can be evaluated through the calculation of the mean, weighted means and Rasch scores and regression analysis [[56], [57], [58]]. Recent studies conducted in literature also shows the use of mean to evaluate the factors of ‘satisfaction” [[59], [60], [61]]. Thus, the mean values will be used to evaluate student satisfaction for this study.

4. Results

This section will present the calculated mean ($\bar{\mathrm{x}}$) for each factor used to derive the user satisfaction for the tool. The standard deviation was also calculated (SD), which indicated the spread of each variable. The responses are measured as follows: “5- Strongly Agree”, “4- Agree”, “3- Neutral”, “2- Disagree” and “1-Strongly Disagree”. For the current article, the overall mean values will be interpreted as follows: “1 and 2- not satisfied”, “3- neutral” and “4 and 5- satisfied”.

• i.Factor 1: Heuristic evaluation of Digital Literacy Scale (DLS)

Table 3 presents users' perception of the heuristic of the digital literacy scale. The mean values for each element are above 4, indicating that the users were satisfied with the heuristics of the digital literacy scale. Heuristic evaluation from the user perspective is necessary to identify the issues that the users may face if the scale is deployed to a wider audience [ [52,60]]. The results reaffirm with the overall mean value of 4.11 that the digital literacy scale is feasible to be used by the greater audience-other students and individuals in the South Pacific and beyond. The SD values are greater than 0.74 and closer to 1, indicating that the responses are similar and there are not many variations.

• ii.Factor 2: Heuristic evaluation of Digital Literacy Intervention (DLIP)

Table 3.

Mean and SD values for factor 1.

Variables	Mean	SD
Appearance/Aesthetics: first impression of the application
The purpose of the DLS is clearly defined	4.16	0.82
The design interface of DLS is simple and easy to understand	4.12	0.82
The DLS has a pleasing color scheme	3.94	0.85
The content is properly organized.	4.18	0.85
The design is consistent	4.20	0.74
The Text and colors are consistent	4.10	0.81
Content: easy for users to find and use the application
The headings are easy to understand	4.20	0.91
The content under each module is easy to understand and attempt	4.14	0.87
The terminologies are easy to understand	4.10	0.88
The result page is easy to understand and interpret	4.00	0.90
The links given is easy to follow	4.20	0.79
Feedback to the user is provided	4.17	0.99
Navigation: makes getting around the application easy
The web pages are easy to navigate through	4.10	1.09
The active page is easy to identify	4.07	0.95
It is easy to navigate back to the home page and result page	4.22	0.77
The website has relevant buttons	4.17	0.79
The organization of information on the website makes sense	4.08	0.84
Overall Mean for Factor 1 = 4.12

Table 4 presents participant perception on the heuristics of the digital literacy intervention program. The intervention consists of user interaction, so heuristic evaluation is important. The mean values for each element are above 4, indicating that the users could use the DLIP, navigate through it, and were satisfied with the appearance and content. For a game-based intervention, the users need to be satisfied with the interface, interaction, usability and gameplay [8,11]. The results from this study confirm that the users were satisfied with the game-based digital literacy intervention. The SD values are closer to 1, thus, the responses are similar to each other.

• iii.Factor 3: Student attitude and perception on the Digital Literacy Tool

Table 4.

Mean and SD values for factor 2.

Variables	Mean	SD
Appearance/Aesthetics: first impression of the application
The purpose of the DLI is clearly defined	4.11	0.85
The design interface of DLI is simple and easy to understand	4.10	0.81
The DLI has a pleasing colour scheme	4.11	0.94
The content is interactive and interesting	4.09	0.87
The design is consistent and user friendly	4.18	0.92
The Text and colors are consistent	4.06	1.05
Content: easy for users to find and use the application
The instructions are easy to understand	4.29	0.96
The content under each module is easy to understand and attempt	4.18	0.87
The terminologies are easy to understand	4.16	0.86
The content is related to the topic	4.01	1.06
The result page is easy to understand and interpret	4.16	0.93
The links given are easy to follow	4.08	1.04
The operations, actions and responses are consistent	4.17	0.87
Feedback to the user is provided	4.14	0.89
Navigation: makes getting around the application easy
The pages are easy to navigate through	4.03	0.97
It is easy to navigate from one page to the next	4.06	1.00
The user has good control of the interface	4.00	1.02
The website has relevant buttons	4.10	0.89
The organization of information on the website makes sense	4.08	1.01
Overall Mean for Factor 2 = 4.11

The user attitude and perception is essential when intervention or a learning method is implemented [11]. Positive attitude and perception of users validate the intervention and its success [62,63]. Table 5 shows the results gathered from the current study. The mean perceived responses of the users are above 4 for all the elements under attitude and perception. The overall mean is 4.25 therefore, the perceived attitude and participation of users are positive for the digital literacy scale and the digital literacy intervention program. The response for perceived usefulness and perceived ease of use for the digital literacy tool is above 4 thus it can be stated that the users found the digital literacy tool useful and easy to use. The SD values are greater than 0.69 as such the responses of the participants are similar.

• iv.Correlation between the three factors (heuristics, student attitude and student perception)

Table 5.

Mean and SD values for factor 2.

	Mean	SD
Student Attitude
Overall, I am satisfied with my experience using the DLT.	4.28	0.74
Attempting the DLT improved my digital literacy skills and knowledge	4.30	0.76
I know I can rely on the DLT to improve my digital literacy skills	4.25	0.74
I find DLT useful and easy to use	4.34	0.72
I know how to use DLT but I feel some form of training can be provided to successfully use this toll	4.29	0.74
DLT is user- friendly and I trust that it can be used to improve the digital literacy of individuals	4.31	0.71
I would encourage others to attempt DLT	4.31	0.72
I feel good about using the DLT	4.28	0.75
I am willing to recommend the DLT to my friends and other people that I know	4.23	0.81
I feel that DLT is one of the best ways one can improve his/her digital skills	4.25	0.77
Perceived Usefulness
The DLT enables me to know my digital literacy competency and then improves my digital literacy skills	4.27	0.70
Using DLT makes me aware of the different literacies that I must possess in this digital age	4.26	0.69
DLT enhances my digital literacy skills and improves the skills I currently possess	4.24	0.77
DLT is effective and useful for a 21st-century student	4.28	0.70
DLT provides sufficient digital literacy content and self-evaluation methods	4.30	0.70
Perceived Ease of Use
DLT is easy to use and easy to learn tool	4.21	0.76
DLT is user friendly	4.23	0.71
A user can control his/her learning while using DLT	4.25	0.74
The content in DLT is easy to remember and options are available to navigate back to the content	4.09	0.86
The navigation in DLT is easy	4.13	0.77
Anyone can learn how to use DLT	4.19	0.79
Overall Mean for Factor 3 = 4.25

A correlation analysis was performed to evaluate the relationship between the three factors. A normality test was carried out first to decide the type of correlation analysis. The results of the Shapiro–Wilk test were as such-sig value = 0.000. Since p < 0.05, it can be stated that the data is not normally distributed. As such the Spearman’s rho for correlation test was performed. The results are presented in Table 6.

Table 6.

Correlation test results.

N = 105	heuristics	student attitude	student perception
heuristics	1.00
student attitude	0.883	1.00
student perception	0.842	0.818	1.00

The results from Table 6 indicate that there is a statistically significant correlation the heuristics of DLIP, student attitude and student perception. Furthermore the results indicate that if the users were satisfied with the heuristics of DLS and DLIP, they have positive attitude and perception about the use of the digital literacy tool- -digilitFJ. Also, user satisfaction with the heuristics of the digital literacy tool leads to a positive perception of the usefulness and usability of the digital literacy tool. Research has shown that the heuristics of an application influence the behavioural intentions and attitude strength of the users. The correlation results affirm the studies conducted as the correlation coefficients are above 0.8. According to Ref. [64], a correlation coefficient between 0.70 and 0.89 indicates a strong correlation.

• v.Effectiveness of the digital literacy tool- -digilitFJ

The reliability and validity of the digital literacy scale and digital literacy intervention program have been tested; readers are referred to Ref. [11]. The details for the statistical test on the effectiveness of the digital literacy tool are also included in the mentioned paper. However, in brief, it has been statistically proven that the digital literacy tool was effective. The Cohen’s d test was performed to evaluate the effectiveness of the digital literacy tool. The calculated Cohen’s d value for overall digital literacy for an individual was 0.473 with a sig value of 0.00, which indicate that there was a moderate impact of the digital literacy tool on the digital literacy of the users. For individual modules, Cohen’s d values were as follows: media literacy −0.51, communication literacy- 0.96, information literacy −0.08, visual literacy −0.37, technology literacy −0.32, computer literacy- 0.55. The impact scale was as follows “0.2 be considered a 'small' effect size, 0.5 represents a 'medium' effect size and 0.8 a 'large' effect size “[64] (McLeod, 2019).

5. Discussion

The objectives of this article were to evaluate user satisfaction, user attitude and effectiveness of the digital literacy tool. The study pursued five research questions.

• i.Are the users satisfied with the heuristics of the digital literacy scale?
• ii.Are the users satisfied with the heuristics of the digital literacy intervention program?

The newly developed digital literacy tool – digital literacy scale (DLS) and digital literacy intervention program (DLIP) was the very first one for the South Pacific and unique in its nature globally. Prior digital literacy scales developed had aspects of essential 21st century skills missing [7,8,39] thus the authors of this paper designed and developed a new digital literacy scale that measured complete digital competencies of individuals. Moreover, DLIP is the very first digital literacy intervention program that can be used to improve digital literacy skills of individiuals in the South Pacific and beyond. Since both DLS and DLIP are new additions, a user satisfaction and evaluation is necessary. Therefore, heuristics evaluation of the DLS and DLIP was successsfully carried out to evaluate their user satisfaction. Literature states that the 21st-century involves a lot of interaction-based learning using digital platforms; hence evaluation of heuristics is critical [36]. A study conducted by Ref. [65] disclosed that heuristics evaluation of game-based context is important to improve user experience and user engagement. For this study, the heuristics of the DLS and DLIP was evaluated using Neilson’s heuristics. The results showed that the users were satisfied with the appearance, content and navigability of the digital literacy tools – DLS and DLIP. The mean values for all the attributes used to test heuristics for the digital literacy scale and digital literacy intervention program were close to 4 and above. The mean values indicated that DLS and DLIP are developed for considering the satisfaction of the users thus any user can easily use the digital literacy tool for enhancement of their digital literacy skills. Moving on, the student attitude and perception was evaluated. Prior studies conducted showed that heuristic satisfaction of an application is directly correlated to the attitude and perception of the users [ [40,65]] (Figueroa et al., 2019; Rajanen & Rajanen, 2018).

• iii.What is the attitude and perception of the users to the digital literacy tool?

For the current study, the results for attitude and perception for the users showed the users have positive attitudes and perceptions about the digital literacy tool-digilitFJ. The mean for the factor “attitude and perception” wass above 4 which indicated that the users have a positive attitude and perception towards the DLS and DLIP. Prior studies conducted showed that the positive attitude and perception of the users led to successful implementation and use of any system [9,66,40]. Since the results for the user attitude and perception is positive, digital literacy tool, if implemented, will be a success as well. Furthermore, a correlation analysis between user satisfaction on the heuristics of DLS and DLIP and student attitude and perception was evaluated.

• iv.Is there any correlation between the three factors– the satisfaction on the heuristics of the digital literacy scale and the digital literacy intervention program and the attitude and perception of the students to the digital literacy tool?

The correlation between the three factors showed that heuristics is a determinant of user attitude and perception. As indicated above, prior studies have shown that users who are satisfied with the heuristics of an application have a positive attitude and perception of the application [9, 66, 40, 41]). The correlation coefficient for this study for the three factors-heuristics, student attitude and student perception was above 0.8, indicating a strong correlation between the factors.

Moreover, the effectiveness of the digital literacy tool was measured using the Cohen’s d test.

• v.Is the digital literacy tool effective?

Cohen’s d value shows whether an intervention shows an impact or not [67] (McLeod, 2019). Studies conducted by Refs. [9, 66, 40, 41, 68] have shown that the Cohen’s d value is the best measure to evaluate the effect of any intervention. As such, it can be stated that the digital literacy tool, which consists of DLS and DLIP was effective since Cohen’s d value was above 0.0473.

The results showed that the digital literacy tool was effective and satisfactory from the user’s perspective.

5.1. Limitations

The sample size and the sample background are limitations of the current study. In the future, it is recommended that the digital literacy tool's effectiveness and user attitude and satisfaction be evaluated with a broader audience-involving students from different levels of education, individuals from the work sector, and individuals in society. Furthermore, since the digital literacy status is self-evaluated, the DLS can be further improved to test the digital literacy of individuals using “skills test questions” at the initial stage instead of self-evaluated questions, or probably include a hybrid questionnaire. Other factors such as emotional, psychological and socio-economic factors need also be included.

6. Conclusion

21st-century learning is associated with technology and is driven by appropriate learning models and tools. As technologies evolve, the learning methodologies and approaches need to be modified to reflect the skills needed to learn with new technologies. As such existing learning frameworks, models and tools also need to be appropriately modified. While the education sector was getting comfortable with the use of technology-enabled teaching and learning for predominantly face to face and blended courses, the outbreak of the Covid-19 pandemic forced the facilitation to move fully online. The teachers and as well as the students were demanded to acquire relevant digital skills to continue teaching and learning during the pandemic. The demand for advocacy and getting accustomed to digital literacy skills gained momentum. A critical analysis of existing models showed that important digital literacy skills were missing thus leading to the existing digital skills gap amongst the facilitators and the learners. Motivated from the above, the current study proposes a validated digital literacy model which can be combined with the existing and future educational models to minimize the existing digital skills gap.

As discussed in the literature review section of this paper, the existing digital literacy and educational frameworks and models had skills gap which became evident during the Covid-19 pandemic. As such, the authors proposed a validated digital literacy model to be incorporated with the existing educational models to curb the issue of the digital skills gap in the South Pacific. The digital literacy model has components that measure and improve the digital literacy skills of individuals and if implemented appropriately, it can reap benefits for the Pacific populace.

The present study evaluated the effectiveness of the proposed digital literacy model in terms of user satisfaction, attitude and the effectiveness of this innovative tool. The results showed that the users were satisfied with the heuristics of the DLS and DLIP thus had a positive attitude to the usability of the digital literacy tool. Further analysis also showed that the digital literacy tool was effective with the effect size or the Cohen’s d value of 0.0473 and above. From the results and discussions in this paper, it can be concluded that the proposed digital literacy model is reliable and is effective to be used with other educational models. Thus the processed digital literacy model is a preamble to educationists to improve their educational framework and pedagogies and reduce the digital skills gap.

Author contribution statement

Pritika Reddy: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Wrote the paper.

Kaylash Chaudhary: Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data.

Shamina Hussein: Conceived and designed the experiments; Performed the experiments; Wrote the paper.

Data availability statement

Data included in article/supp. Material/referenced in article.

Declaration of interest's statement

The authors declare no conflict of interest.

Appendix ASupplementary data

The following is the Supplementary data to this article.