Validation of the DeLone and McLean Information Systems Success Model

Abstract

Objectives

This study is an adaptation of the widely used DeLone and McLean information system success model in the context of hospital information systems in a developing country.

Methods

A survey research design was adopted in the study. A structured questionnaire was used to collect data from 442 health information management personnel in five Nigerian teaching hospitals. A structural equation modeling technique was used to validate the model's constructs.

Results

It was revealed that system quality significantly influenced use (β = 0.53, p < 0.001) and user satisfaction (β = 0.17, p < 0.001). Information quality significantly influenced use (β = 0.24, p < 0.001) and user satisfaction (β = 0.17, p < 0.001). Also, service quality significantly influenced use (β = 0.22, p < 0.001) and user satisfaction (β = 0.51, p < 0.001). However, use did not significantly influence user satisfaction (β = 0.00, p > 0.05), but it significantly influenced perceived net benefits (β = 0.21, p < 0.001). Furthermore, user satisfaction did not significantly influence perceived net benefits (β = 0.00, p > 0.05).

Conclusions

The study validates the DeLone and McLean information system success model in the context of a hospital information system in a developing country. Importantly, system quality and use were found to be important measures of hospital information system success. It is, therefore, imperative that hospital information systems are designed in such ways that are easy to use, flexible, and functional to serve their purpose.

I. Introduction

The health of a nation's citizenry is to a large extent dependent on the quality of health services rendered by its hospitals. Hospitals in developing countries, including Nigeria, face a number of challenges. Key among these challenges is the management of patients' medical records and the overall hospital information. Observation has revealed that a number of hospitals in the Nigeria experience issues such as misplacement and sometimes duplication of patient records, unorganized records in an array of forms, unstructured data, and so on, which have negative consequences on the quality of healthcare delivery.

To overcome these challenges and improve the general quality of healthcare, the implementation of an electronic hospital information systems is crucial. It is noteworthy, however, that implementing electronic hospital information systems requires huge capital investment, usually with a high expectation of returns [1]. Therefore, it becomes expedient to evaluate the success of such systems, considering that investing hugely in IT projects does not necessarily guarantee success [2].

Understanding information system success is an ongoing area of interest not only to researchers but also to practitioners and management stakeholders. Such understanding helps highlight the value of the system and can serve as a basis for subsequent decisions regarding such systems. There are various approaches to assessing the success of information systems (IS). The popular and most validated measure is the DeLone and McLean IS success model (D&M model). The D&M model was first propounded in 1992 and was updated with some modifications in 2003 [3,4]. This paper focuses on the measures provided in the updated model. The model provides six interrelated dimensions of IS success: system quality, information quality, service quality, (intention to) use, user satisfaction, and net benefits.

The D&M model has been applied and validated in a number of IS studies. For instance, it has been tested and validated in studies assessing the success of e-commerce systems [5,6], knowledge management systems [7], e-government systems [8], and much more [9,10]. In particular, the model has also been tested and applied in a number of health information systems studies in both developed and developing countries—as Bossen et al. [1] pointed out that the D&M model serves as a good framework for evaluating health information systems. For instance, in modeling the antecedents of Electronic Medical Record (EMR) system implementation success in Ethiopian hospitals, it was found that the constructs and relationships depicted in the D&M model are applicable in assessing the success of a system [11]. Another study confirmed the model as applicable to healthcare settings when it was used as a framework for investigating mobile device usage within healthcare and found corresponding relations among the variables [12]. Also, a study conducted by Cho et al. [13] evaluated the performance of newly developed information systems in three public hospitals in Korea based on the D&M model. They reported that the quality dimensions (information, system, and service quality) significantly related to user satisfaction and net benefits, thereby upholding the model [13]. In the same vein, a study [14] of an emergency response medical information system confirmed the usefulness of the D&M model in evaluating health information systems. It was found that the quality dimensions depicted in the model influenced user satisfaction and intention to use; which consequently affected the individuals' perception of the system's impact [14].

It is evident that the D&M model can be used to assess the success of hospital information systems. It is pertinent to mention that most of the studies assessing success based on the D&M model have been carried out in developed countries, with few explicitly validating the model in the context of hospital information systems in a developing country. The major objective of this study is therefore to validate the updated D&M model for assessing the success of hospital information systems in a developing country.

II. Methods

1. Research Model

This study is guided by the updated D&M model [4]. The model offers six interrelated constructs of information systems success measures, that is, the quality dimensions (information, system, and service quality) which could affect subsequent use or intention to use and user satisfaction. It is also suggested that some benefits tagged as net benefits will be achieved as a result of use and/or user satisfaction. These net benefits could consequently affect user satisfaction and continued use of the system. The constructs and their operationalization in this study are further explained thus:

• System quality: This is said to measure the desirable characteristics of an information system. Several IS studies have measured this using such characteristics as perceived ease of use, system features, response time, and flexibility [9]. However, this study assessed system quality by examining the ease of use associated with hospital information systems as well as functionality and flexibility.

• Information quality: This has to do with content issues and characteristics of the information systems output. It has been measured by examining the output of an information system in terms of timeliness, accuracy, reliability, and trustworthiness [9]. This study operationalized information quality in terms of the correctness, usefulness and timeliness of the information generated by the hospital information system in use.

• Service quality: This is measured in terms of the quality of support rendered by the information system's developer. Studies have assessed this using service quality dimensions, such as assurance and responsiveness by the systems support department, as well as the provision of user training [9]. In this study, service quality was measured by examining the technical support in place for users of a hospital information system, the network infrastructure in place, and the reliability of the system.

• Intention to use/Use: This is concerned with assessing the manner in which an information system is used. Various studies have measured this by examining actual usage or sometimes, the frequency of use [9]. The intention to use is said to be an alternative measure to use for some other context, depending on the nature of the information system [4]. Since most institutions having a hospital information system in place expect and mandate their employees to use the system, this study assessed use from the angle of perceived usefulness, as studies have pointed out measuring actual use, in this case, may be pointless [11].

• User satisfaction: This is considered one of the most important measures of systems success, often measured by overall user satisfaction [4,9]. It was assessed in the study by capturing overall user satisfaction with the hospital information system.

• Net benefits: This is also regarded as one of the most important measures of IS success, and it constitutes the extent to which an IS contributes to the success of various stakeholders, whether positive or negative. It has been measured by sometimes assessing individual impact or organizational impact [9]. This study, however, examined net benefits as perceived net benefits, since primary users of the hospital information system were surveyed in this study.

The model guiding this study is presented in Figure 1. As such the following hypotheses were postulated and tested:

Figure 1. Research model.

H1: System quality will positively affect use.

H2: System quality will positively affect user satisfaction.

H3: Information quality will positively affect use.

H4: Information quality will positively affect user satisfaction.

H5: Service quality will positively affect use.

H6: Service quality will positively affect user satisfaction.

H7: Use will positively affect user satisfaction.

H8: Use will positively affect perceived net benefit.

H9: User satisfaction will positively affect perceived net benefit.

2. Study Design and Setting

A survey design was adopted for this study. The population of study comprised 442 health information management (HIM) personnel who were the primary users of the hospital information system in five Nigerian teaching hospitals that have implemented a form of electronic hospital information system. It is worth mentioning that electronic hospital information system implementation is still in its infancy in most hospitals in Nigeria; therefore, the primary users are the HIM personnel who are in charge of managing patients' records. Thus, 442 HIM personnel served as respondents in this study.

3. Instrument

A questionnaire was developed for the purpose of gathering the data needed for analysis. To ensure validity, questions assessing each construct were adapted from prior studies with validated scales [11,13,15,16,17]. In all, a 24-item questionnaire was developed. The questionnaire was given to subject experts for the purpose of adjudging face and content validity. Adjustments were made where necessary before copies of the questionnaire were distributed to participants who gave their consent. A total of 375 copies of the returned questionnaire were found useful for analysis.

4. Method of Data Analysis

A structural equation modeling (SEM) technique was used to examine the extent of relationships between the constructs of the model. This was done using the AMOS ver. 20.0 software (IBM, Armonk, NY, USA). First, a confirmatory factor analysis was run to establish the model's goodness-of-fit. After that, the psychometric properties of the model were evaluated in terms of composite reliability and convergent validity. Thereafter, the paths coefficients of the model were examined.

III. Results

1. Model Fit

To assess the model's goodness-of-fit, the following six common model-fit measures were adopted: the chi-square ratio, goodness-of-fit index (GFI), adjusted goodness-of-fit index (AGFI), normalized fit index (NFI), root mean square residual (RMSR), and root mean square error of approximation (RMSEA). The result in Table 1 shows the model fit indices exceeded their recommended acceptance levels [18,19], thereby indicating the measurement model has a fairly good fit with the data collected.

Table 1. Model fit indices.

GFI: goodness-of-fit index, AGFI: adjusted goodness-of-fit index, NFI: normalized fit index, CFI: comparative fit index, RMSR: root mean square residual, RMSEA: root mean square error of approximation.

2. Validity and Reliability

The model was examined for the reliability, construct reliability, and convergent validity of individual items. The individual item reliability was deemed adequate with a standard loading greater than 0.7 [20]. The results shown in Table 2 indicate that all items possessed standard loadings greater than 0.7. Construct reliability was assessed using composite reliability (CR) as a measure of internal consistency, while the convergent validity was assessed by determining the average variance extracted (AVE). These were calculated using the Stat Tools Package developed by Gaskin [21]. The results shown in Table 2 indicate that the model is reliable and has acceptable convergent validity, as the CR and AVE are beyond 0.7 and 0.5, respectively, as recommended by Nunnally and Bernstein [22].

Table 2. Composite reliability and average variance extracted.

HIS: hospital information system, SD: standard loadings, CR: composite reliability, AVE: average variance extracted.

3. Test of Hypotheses

The SEM analysis provided the standardized path coefficients, p-values, and variance explained by the endogenous variables which were used in establishing the extent of the relationships existing among the variables. The results are shown in the structural model in Figure 2 and Table 3. The result indicate that, except H7 and H9, other hypotheses were significantly supported. System quality was found to significantly influence use (β = 0.53, p < 0.001) and user satisfaction (β = 0.17, p < 0.001). Information quality significantly influenced use (β = 0.24, p < 0.001) and user Satisfaction (β = 0.17, p < 0.001). Also, service quality significantly influenced use (β = 0.22, p < 0.001) and user satisfaction (β = 0.51, p < 0.001). However, while use did not significantly influence user satisfaction (β = 0.00, p > 0.05), it significantly influenced perceived net benefits (β = 0.21, p < 0.001). Furthermore, user satisfaction did not significantly influence perceived net benefits (β = 0.00, p > 0.05). The structural model further shows that use and user satisfaction together account for about 6% of the total variance in perceived net benefits, with use exerting a stronger direct influence than user satisfaction. Also, the quality constructs (system, information, and service quality) together account for about 31% of the variance in use, and about 65% of the variance in user satisfaction.

Figure 2. Structural equation model.

Table 3. Path coefficients.

IV. Discussion

This study validates the updated D&M IS success model in the context of hospital information systems in Nigerian teaching hospitals. Findings from this study give credence to the model's constructs and interrelationships, except the influence of use on user satisfaction and that of user satisfaction on perceived net benefits. However, since the constructs possess valid psychometric properties, and the model shows an acceptable goodness-of-fit, we can infer that this model and its constructs are effective to measure hospital information system success.

Specifically, the study revealed that system quality significantly influenced the use of the hospital information system. This finding is in line with the results of some previous studies [13,14,16]. However, this finding is not in line with some other studies [6,8,23] that did not establish the influence of system quality on the use of an information system. Furthermore, information quality significantly influenced the use of the hospital information system. This result is consistent with those of similar studies [6,8,11,13,16,23]. Also, service quality significantly influenced the use of the hospital information system. While some previous studies have reported similar findings [8,11,23], others have not [6,13,14]. Findings from this study further revealed that system quality had the most influence on the system's use. This implies that, for hospitals to encourage the continued use of implemented information systems, the system quality, in terms of ease of use, flexibility, and functionality, must be given utmost attention. This is particularly important, considering that use had a more significant influence on perceived net benefits than user satisfaction. Also, the quality dimensions (system, information, and service quality) all significantly influenced users' satisfaction with the hospital information system. This finding is in line with the findings of previous studies carried out in similar contexts [11,13,23]. Also, service quality was shown to have the most influence on user satisfaction. However, user satisfaction was not seen to predict perceived benefits. Furthermore, there was no significant influence of use on user satisfaction with the hospital information system. Therefore, it follows that for a hospital information system to attain success, as perceived via net benefits, the system must be in continued use. Consequently, for continued use to be achieved, more attention should be paid to system quality.

The findings from this study run contrary to similar studies in the healthcare sector that have called for more focus on service quality to improve user satisfaction and consequently achieve systems success [11,13,23]. The type and context of the system in use may be a factor here. In addition, a situation whereby users are not incentivized to use the system but are mandated to do so may have warranted a need to improve service quality and satisfaction. Another point of divergence noted in this study is the lack of influence of use on user satisfaction, as found in similar studies [11,13,23]. However, the findings of this study support similar studies that have reported that system quality and use are important measures of health-related information systems success [14,24].

In conclusion, following the recommendation of DeLone and McLean to continuously test and adapt their model in different contexts, this study has validated the model in the context of hospital information systems in Nigerian teaching hospitals. Arguably, this study has exclusively validated the model in the context of teaching hospitals in a developing country. This study has shown system use to be an important measure of perceived net benefits, which ultimately drive success. Furthermore, while all quality dimensions predicted system use, system quality has the most influence. Therefore, it follows that hospital administrators and information system developers in a developing country like Nigeria should pay more attention to system quality characteristics, such as the ease of use, functionality, and flexibility.

While it does not undermine the results of this study, there was a limitation in terms of the category of respondents surveyed. Subsequent studies should survey all users of a hospital information system once it is in a steady state of use. Also, adopting a mixed methods approach may help to gain deeper insights into hospital information system success.