Skip to main content
NCBI home page
Health Systems logoLink to Health Systems
. 2023 Jan 8;12(1):85–97. doi: 10.1080/20476965.2022.2162445

Assessing the impact of software quality models in healthcare software systems

Elisabetta Ronchieri 1,✉, Marco Canaparo 1
PMCID: PMC10013535  PMID: 36926376

ABSTRACT

Health and medical care are safety-critical environments. The development of a safe and reliable healthcare software system is one of the most important processes which may affect the accuracy of clinical decision making and have an impact on determining diagnoses of and treatment for diseases. The aim of this study is to assess if software quality models, such as McCall, can have an impact on healthcare software quality for the purpose of reducing software failures. We have identified the main criticalities of healthcare software systems based on existing literature, software quality models and programming language standards; we have related these criticalities to software characteristics; we have analysed the relationship between the main criticalities of healthcare software systems and their software characteristics. The following criticalities - the risk of changing hard-coded software and the importance of developing maintainable software – must be addressed at the start of the software development process.

KEYWORDS: Software quality models, healthcare software systems, software quality

1. Introduction

Healthcare organisations are increasingly using complex software applications to manage an abundance of medical data and their processing, in order to make an early and accurate diagnosis and thus support clinical decision making. Due to the importance of these software applications, healthcare software developers must validate and verify their software to guarantee its maintainability, reliability and safety characteristics, that is its quality. Software malfunctions, such as faulty programming and erroneous calculation, are not desirable in any problem domain, but in healthcare environments, they can lead to wrong clinical decision making and consequently may cause death. In other words, healthcare software is different from other types of software, because it may cause threats to and may damage human lives (Aguiar et al., 2010; Ali et al., 2017; Lee et al., 2006; Weber-Jahnke et al., 2013). Furthermore, this type of software envelopes constraints on software quality attributes, such as performance and security (Aguiar et al., 2010), and thus places software quality requirements in the focus of software researchers and healthcare practitioners.

In the last three decades, industry, various organisations for standards and computer scientists have given many definitions of software quality. In 1990, IEEE defined “software quality as the degree to which a system, component, or process meets specified requirements or customer or user needs or expectations”. However, this definition does not explain how to measure software quality or define quality characteristic. In 1996, Fenton and Pfleeger (Fenton and Pfleeger, 1996) stated that “measurement is the process of empirically and objectively assigning numbers or symbols to the attributes of entities in the real world in such a way as to describe them”. The mentioned definition of software quality is still used in literature (Pau, 2020) and included in ISO/IEC, the most recognised standard software.

Software quality models (a) relate software quality to a set of software characteristics and (b) provide software metrics (to be measured) in order to obtain the information needed by developers for technical decisions. In 1990, IEEE defined “software metrics as the quantitative measure of the degree to which a system, component or process possesses a given software attribute”. The various quality models defined over time include different factors (also called characteristics), many of which have similar definitions (Wagner, 2013). Consequently, it is important to define a common terminology, as well as to provide a common definition for the various factors, in order to facilitate the adoption of software quality models in the healthcare software solutions.

Healthcare software developers can validate and verify their software by using existing software quality models (Aghazadeh et al., 2015; Al-Qutaish, 2010). Health authorities and developers of health information technology consider the quality of healthcare software of paramount importance (Hemmati et al., 2017; Richardson et al., 2016). A good example is given by the need of maintaining health information systems over time due to health-specific changes in environments, requirements and functional specifications. These different aspects are also applicable to other contexts, such as banking, aeronautical and aerospace systems. Nevertheless, in the case of healthcare software, potential problems in the code could result in threats to life. It is a known fact that software maintenance tasks alone may usually constitute 70–80% of the total development cost (Sommerville, 2000). In fact, the main issue in software maintenance arises from changes in (or introduction of new) software requirements that cause a repetition of the whole software development cycle from redesign to redeployment. Poor software maintainability may have negative effects on the reliability of changed software and cause a delay to deliver healthcare software solutions to market (Krogstie, 2006). Aghazadeh et al. (2015) examined the effect of these characteristics on healthcare quality indicators (Smith et al., 2005), such as user satisfaction, quality of patient care, and clinical workflow and efficiency. In addition, Garcés et al. in Garcés et al. (2017) provided a detailed panorama on software quality attributes considered to be important in the ambient assistant living domain. However, to the best of our knowledge, there is a lack of a complete, detailed study on how software quality characteristics are being addressed in healthcare software.

The notion of software architecture (Bengtsson & Bosch, 1999; Kazman et al., 2020a, 2020b) has a strong impact on all software qualities and, consequently, on software maintenance. In this context, a set of software activities have been specified by Pigoski in Pigoski (1996). The author recommended that during the software maintenance process a detailed knowledge of the code’s structure is needed in order to carry out impact analysis of an existing or new software requirement. They also may contribute to other activities, such as configuration management, verification and validation, quality assurance, reviews, audits and conducting user training. These recommendations are applicable to all types of software across domains to improve their development process and reduce the consequences of any software failure risk.

With the term criticality we refer to software areas or components where we may have a problem (Erbert & Liedtke, 1995). The aim of this study is to map software quality characteristics with healthcare software criticalities to determine which software quality models could be more suitable for healthcare software criticalities. Furthermore, we want to detect healthcare software criticalities that require attention during the software development process (Sommerville, 2000; Stahl et al., 2006).

Based on the aim of this study, three main research questions (RQ) were formulated:

RQ1. Which healthcare software criticalities have to be addressed during the healthcare software development process?

RQ2. What are the most important software characteristics that can be considered for healthcare software development?

RQ3. What is the most important software quality model that can be used by the healthcare software developers to fulfil software quality requirements?

In our view, this study might lead to an improvement in the assessment and treatment of diseases. It provides useful recommendations to enhance software quality and, therefore, reduce the risks faced by patients. This work has an ambitious goal because its applicability requires a strict collaboration between software healthcare companies and researchers in software engineering.

The remainder of this paper is organised as follows. Section 2 is about the methodology we adopted for this research. Section 3 describes criticalities in healthcare software systems. Section 4 summarises the software characteristics extracted from the most known software quality models in existing literature. Section 5 introduces the need of a common terminology for software characteristics. Section 6 describes the way we link healthcare criticalities with software characteristics. Section 7 shows the relationship between healthcare software criticalities and software quality models and provides our answers to the research questions. Finally, Section 8 summarises the paper and draws our conclusions.

2. Research methodology

In this section, we describe the methodology that we have designed for answering our research questions. Our approach was based on our experience done in European Projects to manage software projects (Orviz Fernández et al., 2020). The methodology is composed of five stages:

Stage 1. Define a set of healthcare software criticalities.

In this stage, we have considered existing literature to identify healthcare software criticalities that occur during the various phases of software development.

Stage 2. Define a set of software characteristics.

In this stage, we have considered existing software quality models to collect software characteristics with their descriptions.

Stage 3. Define a common terminology of the software characteristics.

In this stage, we have introduced a common terminology for software characteristics, specifying synonyms to reduce the set of considered software characteristics. Furthermore, we have determined how the various software characteristics are shared in the selected software quality models by using a truth table.

Stage 4. Match each software characteristic with each healthcare criticality.

In this stage, we have identified the set of software characteristics that can be taken into consideration to address the various healthcare software criticalities by using a truth table.

Stage 5. Answer research questions.

In this stage, we have discussed our research questions. We have decided to use a truth table to express our consideration on the set of healthcare software criticalities and the set of software characteristics, built during the first two stages.

The various stages are detailed in the following sections.

3. Criticalities in healthcare software systems – stage 1

Healthcare is a very complex domain where we cannot consider software requirements stable nor extract them at the beginning of the development phase. Both the body of knowledge and medical practices can change from time to time and place to place across different organisations and jurisdictions (Beale, 2005; Paech & Wetter, 2008; Rector, 1999). The size and complexity of the medical domain and the variability of practice on using specific applications pose great challenges for building and maintaining health information systems (Plsek & Greenhalgh, 2001). Rapid advances in the body of knowledge and medical technologies create a need to modify health information systems (Beale, 2005; Pronovost et al., 2009; Rector, 1999).

Table 1 shows the most common criticalities in healthcare software systems. It specifies an identifier, a short title, a description and a recommended action to fulfil or solution to adopt. While the title can be applicable to different types of software, the description is mainly related to healthcare software, and the recommended action or solution is not context specific. Table 1 describes, in an innovative way, detailed general criticalities in the context of the medical area.

Table 1.

The most common criticalities in healthcare software systems.

Identifier Title Description Action/Solution
C1 Maintainability comparison between software products of the same healthcare domain During a software lifetime, clinical and business requirements may change and increase in number introducing many modifications, which may raise the cost of maintenance. Furthermore, domain knowledge can cause code changes leading to a change in the software developed. To avoid the same problem in the future it is recommended to carry out a comparison between software programs as concern their maintainability. Atalag et al. (2014), for instance, compared software maintainability between GastrOS and OST in the gastrointestinal endoscopy domain by measuring time and software size changes.
C2 Software maintainability in the presence of embedded knowledge Nowadays, the widespread approach towards software development is to embedding domain knowledge into program and data model as well as to hard-coding approach into software. These approaches can cause the rising of cost during the maintenance phase (Ashton, 2006). To avoid that developers waste time to substantially modify the existing software, it would be necessary to gather configuration settings externally to the application.
C3 The ease to satisfy new software requirements During software life cycle new requirements may be specified. These may be in contradiction to what has been already agreed. It would be necessary to reveal to what extent a specific implementation favours changes that satisfy existing or new requirements.
C4 To support interoperability standards Healthcare applications have to interface with external systems that can provide complex interfaces. Therefore, standards (such as DICOM2) have been developed to simplify their communication. Nevertheless, implementing code that adheres to the standards may require a lot of time and effort. Therefore, it is essential to carefully plan the design of code in order to integrate and implement interoperability standards.
C5 To guarantee reliability One of the most essential requirements of medical software systems is reliability. This feature is affected by size and complexity of software (Khatri et al., 2012; Sharma & Dubey, 2015) that are growing dramatically in the medical field with the rising of medical technologies. Nowadays, we can hardly find applications with fewer than hundreds of millions of lines of code. Provided that the need of automatic procedures has been increasing in the last few years, medical systems have also increased in their complexity. It would be necessary to introduce design methodologies that makes fault-free software, and/or to remove faults after the development phase and finally to make provisions for the system to operate correctly even in presence of faults (Son & Koo, 2009).
C6 To perform complete unit testing during development life cycle Traditionally developers perform some unit testing throughout the development process, leaving the bulk of the testing effort near the project delivery. Albeit this is the most common method for developing and delivering products, it may lead to serious problems. In first place, it is difficult to estimate the time and effort to remove all defects once a product is completed causing shipping delays. In second place, it may be extremely costly to find problems at the end of a project if fixing them means reworking or redesigning parts of the application. In our experience, as soon as a unit testing procedure is integrated in the development process, requested cost time and effort to set it up decrease enormously. Murphy et al. (2011), for instance, detailed the importance of testing to search errors in the implementation.
C7 To provide reasonable documentations Maintaining software code is a demanding task that needs both software development skills and technical know-how to be carried out. However, many times this activity is conducted without appropriate documents penalising developers for the understanding of the old program code. This critical factor leads to the necessity of standardising maintenance models, giving a predominant role to the writing of documentations.
C8 To validate the medical product Validation is a process responsible for answering the question “Are we building the right product”? It is the principal means of avoiding defects after production and distribution. Validation also defines to what extent a model, simulation and their associated data are a proper representation of the real world from the perspective of the intended users (FDA, 2002). Developers mainly have to regularly check that the product designed satisfies the intended use.
C9 To predict dangerous module Developers are in charge of implementing functionalities or modifying existing ones reducing as much as possible software faults. This is a complex and complicated process since mistakes can be inserted in the code at any time: at the point of release; during modification; whenever changes are made. Software metrics may help finding out faults insertion phenomena by measuring code characteristics. For example, a large software system consists of various modules written in a specific language, whose features are measured by software metrics tools. It would be useful to identify dangerous modules by employing prediction models based on these measurements. Ronchieri and Pia (2018), for example, showed the assessment of the quality of Geant4 and ROOT software by using software metrics and properly statistical methods.
C10 To apply coding standards for developments The number of defects is related to the degree of quality of software. For example, 80% of C++/C software defects are attributable to issues with 20% of the language constructs. To improve quality developers should write code strictly adhering to standards, such as MISRA-C (2004) and MISRA-C++ (2008). Furthermore, standards help developers to avoid language constructs that can lead to exploitable vulnerabilities. Following standards’ code style might be useful to guarantee maintainability over time.
C11 To minimise the overhead The development team is usually composed by developers with different background and experiences. Automating peer review gives many advantages to team developers, such as improving quality of software and reducing costs.
C12 To ensure unrestricted portability on any computer or laptop During software utilisation it might be necessary to change the framework where a program runs. It would be necessary to choose a programming language to favour portability of application on different operating system.
C13 To prevent unauthorised access to data Healthcare software manages sensitive data. Therefore, developers have to include authorisation and authentication policies in their applications in order to avoid deliberate or accidental access to data.
Open in a new tab

On the basis of the existing literature concerning healthcare (such as Ashton in 2006, Atalag et al. in 2014; FDA in 2002), we have searched for issues occurring during the various phases of the software life cycle that match characteristics (or factors) specified by software quality models (such as ISO 2001, IEEE 1990, and McCall et al. 1977) and well-known programming language standards (such as MISRA-C in 2004 and MISRA-C++ in 2008).

4. Software characteristics from software quality models – stage 2

A software quality model contains several quality characteristics (or factors) and criteria against those factors. By using them the quality of a product can be addressed. In this section, we are going to describe the most popular quality models from the very beginning until the recent ones (1977–2011) providing a summary table whenever possible. We have considered the information structure detailed by Berander et al. (2005). Table 2 shows for each quality model three columns: the left one covers attributes; the middle column includes the factors contained in the described quality models; the third column reports the description of each factor.

Table 2.

Quality models factors.

Attribute Factor Description
McCall’s quality model
Quality Correctness The extent to which a program fulfils its specification.
Efficiency The use of resources, e.g., processor time, and storage. It is further categorised into execution efficiency and storage efficiency.
Integrity The protection of the program from unauthorised access.
Reliability The system’s ability not to fail.
Usability The ease of use of software.
Flexibility The ease of making changes required by changes in the operating environment.
Maintainability The effort required to locate and fix a fault in the program within its operating environment.
Testability The ease of testing the program, to ensure that it is error-free and meets its specification.
Interoperability The ease of testing the program, to ensure that it is error-free and meets its specification.
Portability The effort required to transfer a program from one environment to another.
Reusability The ease of reusing software in a different context.
Boehm’s quality model
Quality As-is utility The ease, reliability and efficiency in the use of the product as it is.
Maintainability The ease to understand, modify and retest the software product.
Portability The possibility of using a software product when the environment has been changed.
Efficiency The fulfilment of the software purpose without waste of resources.
Flexibility The ease of incorporating changes.
Reliability The satisfaction of the software functions.
Testability The ease of verifying and evaluating software through a set of criteria.
Understandability The extent to which a program is clear to the inspector.
Usability The extent to which a program is reliable, efficient and human-engineered.
FURPS/FURPS+’s quality model
Functional Functionality The set of features, capabilities and security.
Non-Functional Performance Conditions on functional requirements such as speed, efficiency, availability, accuracy, throughput, response time, recovery time, and resource usage.
Reliability Frequency and severity of failure, recoverability, predictability, accuracy, and mean time between failure (MTBF).
Supportability Testability, extensibility, adaptability, maintainability, compatibility, configurability, serviceability, installability, localizability (internationalisation).
Usability May include human factors, aesthetics, consistency in the user interface, online and context sensitive help, wizards and agents, user documentation, and training materials.
Dromey’s quality model
Quality Efficiency The extent that code fulfils its purpose without waste of resources.
Functionality The existence of a set of functions and their specified properties.
Maintainability The ability of the system to undergo changes with a degree of ease.
Portability The ability of the system or part of the system to move across environments.
Reliability The ability of a system to remain operational over time.
Reusability The usage of system’s components developed by different system with different functionality.
Usability The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.
ISO/IEC 9126’s quality model
Quality Efficiency The relationship between the level of performance of the software and the amount of resources used, under stated conditions.
Functionality A set of attributes that relate to the existence of a set of functions and their specified properties.
Maintainability The effort needed to make specified modifications.
Portability The ability of software to be transferred from one environment to another.
Reliability The capability of software to maintain its level of performance under stated conditions for a stated period.
Usability The effort needed for use, and on the individual assessment of such use, by a stated or implied set of users.
ISO/IEC 25,010’s quality model
Quality Compatibility The degree to which a product, system or component can exchange information with other products, systems or components, and/or perform its required functions, while sharing the same hardware or software environment.
Functional suitability The degree to which a product or system provides functions that meet stated and implied needs when used under specified conditions.
Maintainability The degree of effectiveness and efficiency with which a product or system can be modified to improve it, correct it or adapt it to changes in environment, and in requirements.
Operability The degree to which a product or system has attributes that make it easy to operate and control.
Performance efficiency The performance relative to the amount of resources used under stated conditions.
Portability The degree of effectiveness and efficiency with which a system, product or component can be transferred from one hardware, software or other operational or usage environment to another.
Reliability The degree to which a system, product or component performs specified functions under specified conditions for a specified period.
Security The degree to which a product or system protects information and data so that persons or other products or systems have the degree of data access appropriate to their types and levels of authorisation.
Usability The degree to which a product or system can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.
Open in a new tab

One of the precursors of the current quality models is McCall’s model of 1977 - also known as the General Electric Model (Kitchenham & Pfleeger, 1996; Marciniak, 2002; McCall et al., 1977). This model stems from the US military, developed for the US Air Force to improve the quality of their software products. McCall bridges the gap between users and developers by focusing on several software quality factors. In this model there are three major perspectives (see left column of Table 2 for McCall’s quality model) for defining and identifying the quality of a software product (Berander et al., 2005): product operations collecting operation characteristics; product revision representing the ability to undergo changes; product transition representing the adaptability to new environments. These perspectives are detailed in a hierarchy of: 11 factors (see middle column of Table 2 for McCall’s quality model) for describing the external view of the software (as viewed by the users), 23 quality criteria for describing the internal view of the software (as viewed by the developers), and metrics for providing a scale and method for measurement.

Another predecessor of the current quality models is Boehm’s model of 1978 (Boehm et al., 1976, 1978). This model is similar to the McCall one since it presents a hierarchical structure of three levels that provide quality information: at high level there are 3 factors that are related to basic requirements of actual use of code; at intermediate level there are 7 factors that represent the qualities expected from a software system; at primitive level characteristics can be related to the intermediate level ones and provide the foundation for defining quality metrics.

The FURPS quality model is presented by Robert Grady in 1992 (Grady, 1992) and extended by Rational Software1 (Jacobson et al., 1999; Kruchten, 2000) - now IBM Rational Software – into FURPS+. It is structured in the same manner as the previous two quality models. FURPS stands for Functionality, Usability, Reliability, Performance and Supportability (details in Table 2). Functionality (F) belongs to the functional category, while the others (URPS) are related to the non-functional category. According to Table 2, all the factors include a set of features, such as functionality with the security attribute, performance with the efficiency attribute, and supportability with the adaptability, compatibility, maintainability and testability attributes.

Dromey’s quality model of 1995 (Dromey, 1995, 1996) is general enough to work for different products. According to Berander et al. (2005), this model focuses on the relationship between the quality characteristics and the sub-characteristics, as well as the connection between software product properties and software quality attributes (i.e., characteristics). Furthermore, it is structured around a 5 step process: 1. select a set of high-level quality attributes for the evaluation; 2. list components and/or modules in your system; 3. identify quality-carrying properties for the components and/or modules; 4. determine how each property effects the quality attributes; 5. evaluate the model and identify weaknesses. The model implementation details seven factors details in Table 2.

IEEE has released several standards, two of which are more similar to the models described previously: IEEE/EIA 12,207.0–1996 (IEEE, 1996), and IEEE Std 1061–1998 - IEEE Standard for software quality metrics methodology.

IEEE/EIA 12,207.0–1996 describes the processes for the following life-cycle: primary processes, which relates to acquisition, supply, development, operation, and maintenance; supporting processes, which relates to documentation, configuration management, quality assurance, verification, validation, joint review, audit, and problem resolution; organisation processes, which relates to management, infrastructure, improvement, and training. IEEE/EIA 12,207.0–1996 has been replaced by IEEE Std 12,207:2008 (IEEE, 2008). This international standard focuses on software life cycle processes by providing a common framework with a well-defined terminology. It includes processes, activities and tasks to be adopted during the development, operation, maintenance and disposal of a software product as well as during its acquisition. Furthermore, this describes a process for defining, controlling and improving software life cycle processes.

IEEE Std 1061 provides a methodology for establishing quality requirements and identifying, implementing, analysing and validating the process and product of software quality metrics.

The ISO (ISO/IEC stands for International Organization for Standardisation and International Electro-Technical Engineering) organisation is responsible for a set of standards, such as the ISO 9126 - Software Product Evaluation: Quality Characteristics and Guidelines for their Use-standard takes part (ISO, 2001). This standard provides the specification and evaluation model for the quality of software product (Al-Qutaish, 2010; Zeiss et al., 2007). The ISO quality model is based on the McCall and Boehm’s ones. It is basically structured in the same way, including functionality as a parameter and identifying both internal and external quality characteristics of software products. ISO 9126 proposes a standard which specifies a set of quality factors for software evaluation as shown in Table 2. All the factors include a set of features, such as usability with the understandability attribute and portability with the adaptability attribute.

The ISO/IEC 25,010 quality model of 2007 (ISO, 2008) updates the ISO 9126 model. One of its main objectives is to guide in the development of software products with the specification and evaluation of quality requirements (see Table 2). All the characteristics include a set of features, such as security with the integrity attribute and maintainability with the reusability and testability attributes.

ISO/IEC/IEEE 15,289:2011 (IEEE, 2011) is based on IEEE Std 12,207–2008 and includes requirements for the identification and planning of the specific information products to be developed and revised during systems and software life cycles and service processes.

IEEE Std 730–2014 (IEEE, 2014) is based on both IEEE Std 12,207–2008 and ISO/IEC/IEEE 15,289:2011 and establishes requirements for initiating, planning, controlling and executing the Software Quality Assurance processes of a software development or maintenance project.

5. Common terminology for software characteristics – stage 3

The selection of quality models to be applied to specific cases is a real challenge. In fact, these models specify software characteristics by giving similar names (such as efficiency and performance efficiency) but different descriptions. Therefore, we have decided to provide a common set of software characteristics with their synonyms that we have obtained by exploiting existing literature, such as ISO (2001), Dromey (1996) and Grady (1992).

Following the factors taken from Table 2, we summarise in Table 3 the terminology and concepts for the main seventeen characteristics together with the corresponding references related to quality models, enabling readers to get a common vocabulary.

Table 3.

Terminology for software quality model characteristics.

Characteristic Synonym Description Reference
Compatibility   The ability of two or more software components to exchange information and/or to perform their required functions
while sharing the same hardware or software.		 (ISO, 2008)
Correctness   The ease with which minor defects can be corrected between major releases while the application or component is in use by its users. (Dromey, 1996)
Efficiency Performance efficiency The degree to which the software product provides appropriate performance, relative to the amount of resources used, under stated conditions. (Dromey, 1996; IEEE, 1990; ISO, 2001, 2008)
Flexibility   The degree to which the code possesses the characteristics modifiability, i.e., the ease of incorporating changes once the nature of the desired change has been determined. (Dromey, 1996)
Functionality Functional suitability The degree to which the software product provides functions that need stated and implied needs when the software is used under specified conditions. (ASQ, 2007; ISO, 2001, 2008)
Integrity   The degree to which the accuracy and completeness of assets are safe guarded. (ISO, 2008)
Interoperability Compatibility The degree to which a software is able to interact with specified systems. (ASQ, 2007; ISO, 2001)
Maintainability   The degree to which the software product can be modified. Modifications may include corrections, improvements or adaptation of the software to changes in environment, and in requirements and functional specification. (ISO, 2001, 2008)
Performance Efficiency The degree to which the software product provides appropriate performance, relative to the amount of resources used, under stated conditions. (ISO, 2008)
Portability   The ease with which a system or component can be transferred from one environment to another (extend hardware
or software environment).		 (ISO, 2001, 2008)
Reliability   The degree to which the software product can maintain a specified level of performance when used under specified
conditions.		 (ISO, 2001, 2008)
Reusability Adaptability The degree to which an asset can be used in more than one software system, or in building other assets. (ISO, 2008)
Security   The protection of system items from accidental or malicious access, use, modification, destruction, or disclosure. (ISO, 2008)
Supportability Support
Adaptability		 The ability to extend the program, adaptability and serviceability.
The ease with which a system can be installed and the ease with which problems can be localised.		 (Grady, 1992; ISO, 2001, 2008)
Testability   The degree to which the software product enables modified software to be validated. (ISO, 2008)
Understandability Appropriateness recognisability The degree to which the software product enables users to recognise whether the software is appropriate for their needs. (ISO, 2001, 2008)
Usability Operability
Ease of use
Understandability		 The degree to which the software product makes it easy for users to operate and control it. (ISO, 2001, 2008)
Open in a new tab

6. Software characteristics vs healthcare criticalities – stage 4

Table 4 shows in a schematic way the software characteristics provided by the aforementioned seven quality models. For each model we have specified both the factors detailed in their description and their synonyms. Furthermore, we have counted the implemented characteristics for every quality model to determine the most complete one.

Table 4.

Quality models comparison: the coloured cells (with the 1 value) indicate matching between characteristic and model. There are reusability and understandability characteristics that belong to all quality models except for Boehm and FURPS respectively. There are three characteristics – i.e., functionality, portability and testability – that belong to all quality models except two: the first one is not included in McCall and Boehm; the second one is not included in FURPS and IEEE; the third one is not included in Dromey and ISO/IEC 9126. There are three characteristics that belong to four models: compatibility is in McCall, FURPS, ISO/IEC 25,010 and IEEE; interoperability is in McCall, FURPS, ISO/IEC 25,010 and IEEE; supportability is in FURPS, ISO/IEC 9126, ISO/IEC 25,010 and IEEE. There are three characteristics that are included in three models: flexibility is included in McCall, Boehm and IEEE; integrity is included in McCall, ISO/IEC 25,010 and IEEE; security belong to FURPS, ISO/IEC 25,010 and IEEE. Finally, correctness belongs to McCall and IEEE.

  McCall Boehm Dromey FURPS ISO/IEC 9126 ISO/IEC 25010 IEEE
Correctness 1 0 0 0 0 0 1
Flexibility 1 1 0 0 0 0 1
Integrity 1 0 0 0 0 1 1
Security 0 0 0 1 0 1 1
Compatibility 1 0 0 1 0 1 1
Interoperability 1 0 0 1 0 1 1
Supportability 0 0 0 1 1 1 1
Functionality 0 0 1 1 1 1 1
Portability 1 1 1 0 1 1 0
Testability 1 1 0 1 0 1 1
Reusability 1 0 1 1 1 1 1
Understandability 1 1 1 0 1 1 1
Efficiency 1 1 1 1 1 1 1
Maintainability 1 1 1 1 1 1 1
Performance 1 1 1 1 1 1 1
Reliability 1 1 1 1 1 1 1
Usability 1 1 1 1 1 1 1
Open in a new tab

As shown in Table 4, efficiency, maintainability, performance, reliability and usability are the ones common to all quality models. From Table 4, we derive that the IEEE model is the one that covers most of the characteristics requested by the health software criticalities. However, it lacks one important feature, such as portability: therefore, all models except FURPS can be used to bridge this gap. Figure 1 shows the number of characteristics that are matched by the described quality models.

Figure 1.

Figure 1.

Open in a new tab

The number of software characteristics over quality models.

Furthermore, we observe that McCall’s quality model measures subjectively the quality based on the judgement on the person(s) answering the questions (yes or no questions); FURPS quality model is built and extended to be used in the IBM Rational Software Company for the benefits of that company; ISO/IEC 9126 quality model has been built based on an international consensus and agreement from all the country members of the ISO organisation; ISO/IEC 25,010 quality model is an update of ISO/IEC 9126 and it redefines the fundamental characteristics increasing them from six to eight. In addition to that, as Al-Qutaish (2010) did, we notice that the lower level of the McCall’s, Boehm’s, Dromey’s and FURPS quality models are neither clearly nor completely defined and connected to the upper level of the quality models.

We also summarise some observations found in the existing literature about the comparison of software quality models. According to Samadhiya et al. (2010), McCall’s quality models may be more useful in a bottom to top approach to software quality; on the contrary, Boehm’s provides users with basic support for a top to bottom approach. More specifically, Boehm’s quality model has a hierarchical structure which presents, at the top, the concerns of end-users; in the bottom, instead, the interests of technically inclined personnel are represented. Therefore, the user perspective of quality is favoured in Boehm’s quality model. Despite looking very similar the two quality models present some differences: on the one hand, McCall’s focuses on the precise measurement of the high-level characteristics, on the other hand Boehm’s is based on a wider set of characteristics which mostly concern maintainability. Moreover, only Boehm’s quality model deals with the model’s effort on software maintenance cost effectiveness since it is the first advantage of employing software quality considerations. According to Awan et al. (2015), FURPS quality model considers only customer perspective for assessing a product efficiency. Some quality features, such as reusability, reliability and maintainability, which typically belong to developer perspective, are ignored. IEEE quality model mostly focuses on the maintenance of software product. It employs iterative approach (Basil & Turner, 1975) for execution and management of software maintenance events. Furthermore, efficiency and functionality of software product are considered.

7. Results and discussions – stage 5

The main outcomes of this work are summarised in Table 5, Figure 2 and Table 6.

Table 5.

Healthcare software criticalities comparison: the coloured cells (with the 1 value) indicate matching between software characteristic and healthcare criticality identifier. For example, C1 (Maintainability comparison between software products of the same healthcare domain) match with the correctness and maintainability characteristics, while C2 (Software maintainability in the presence of embedded knowledge) matches with the correctness, supportability and maintainability characteristics. See table 1 to look at the meaning of the healthcare criticalities identifiers.

  C1 C2 C3 C4 C5 C6 C7 C8 C9 C10 C11 C12 C13
Correctness 1 1 0 0 0 1 0 0 0 0 0 0 0
Flexibility 0 0 1 0 0 0 0 0 0 0 0 0 0
Integrity 0 0 0 0 1 0 0 0 0 0 0 0 0
Compatibility 0 0 0 1 0 0 0 0 0 0 0 1 0
Security 0 0 0 0 0 0 0 0 0 0 0 0 1
Understandability 0 0 0 0 0 0 1 0 0 0 0 0 0
Interoperability 0 0 0 1 0 0 0 0 0 0 0 0 0
Supportability 0 1 0 0 0 0 0 0 0 0 0 0 0
Portability 0 0 0 0 0 0 0 0 0 0 0 1 0
Reusability 0 0 1 0 0 0 0 0 0 0 0 0 0
Testability 0 0 0 0 0 1 0 1 0 0 0 0 0
Efficiency 0 0 0 0 0 0 0 0 0 0 1 0 0
Maintainability 1 1 1 0 0 0 0 0 0 1 0 0 0
Reliability 0 0 0 0 1 0 0 0 1 0 0 0 0
Usability 0 0 0 0 0 0 1 1 0 0 0 0 0
Open in a new tab

Figure 2.

Figure 2.

Open in a new tab

The number of software characteristics over healthcare software criticalities.

Table 6.

Healthcare software criticalities over number of software characteristics. C2 and C3 are the healthcare software criticalities that require particular attention: the one that highlights the risk of changing hard-coded software and the one that highlights the importance of developing software that can be maintained easily.

3 Characteristics 2 Characteristics 1 Characteristic
C2 (Software maintainability in the presence of embedded knowledge)
C3 (The ease to satisfy new software requirements)		 C1 (Maintainability comparison between software products of the same healthcare domain)
C4 (To support interoperability standards)
C5 (To guarantee reliability)
C6 (To perform complete unit testing during development life cycle)
C7 (To provide reasonable documentation)
C8 (To validate the medical product)
C11 (To minimise the overhead)
C12 (To ensure unrestricted portability on any computer or laptop)		 C9 (To predict dangerous module)
C10 (To apply coding standards for developments)
C13 (To prevent unauthorised access to data)
Open in a new tab

Table 5 shows the matches between each healthcare criticality and software characteristic. Combining information of healthcare criticalities and software characteristics, we were able to identify the set of factors that can be necessary in order to address a criticality. Both functionality and performance are related to specific products: thus, we excluded them from Table 5. Healthcare software providers have to use adequate instruments (like customised dashboard) to monitor all the factors summarised in Table 5. Figure 2 details the number of software characteristics over the healthcare software criticalities.

Table 6 shows the healthcare software criticalities grouped according to the same number of software characteristics. The healthcare software criticalities that require particular attention are C2 and C3, the one that highlights the risk of changing hard-coded software and the one that highlights the importance of developing software that can be maintained easily. C2 needs to satisfy the following factors: correctness, supportability and maintainability. On the other hand, C3 requires to satisfy the following factors: flexibility, reusability and maintainability. The other criticalities require to satisfy one or two characteristics. While most factors need to be addressed by one or two criticalities, this is not true for the maintainability characteristic that is requested by C1, C2, C3 and C10 healthcare software criticalities: a possible explanation is the fact that those criticalities focus on the changes to be done in the software after its delivery and production phases.

We have conducted this research formulating a set of research questions. In the following, we discuss our findings concerning research questions, highlighting evidence gathered from the process.

RQ1. Which healthcare software criticalities have to be addressed during the software development process?

Our study identified 13 healthcare software criticalities that are related to healthcare software. A remarkable number of criticalities concerns issues with maintainability that includes the need to understand the differences amongst different software, the necessity to avoid embedding knowledge, the satisfaction of new software requirements, and the need to apply coding standards. Two criticalities regard the need to guarantee compatibility, reliability, testability and usability. One criticality regards the need to support correctness, efficiency, flexibility, integrity, interoperability, portability, reusability, security, supportability, and understandability. Furthermore, C2 and C3 are those criticalities that require attention affecting the highest number of characteristics: C2 is about the risk of changing hard-coded software; C3 focuses on the importance of developing maintainable software.

RQ2. What are the most important software characteristics that can be considered for healthcare software development?

The most important characteristic to be considered in healthcare is maintainability, covering 4 healthcare software criticalities C1, C2, C3, C10 (see Table 1 for their descriptions) as shown in Table 5. The compatibility, reliability, testability and usability characteristics follow the maintainability one with 2 criticalities each (see Table 5 to identify the various criticalities). These software characteristics can be measured by using proper software metrics and adequate tools. Tracking and analysing their values represent ways to determine and improve the quality of software, and to predict the quality while the software development project is still ongoing. In literature, there are numerous metrics that, despite having similar names, are often defined and implemented in different ways; therefore, it is difficult to compare software code by using a given metric without knowing its exact definition, the methods by which it is calculated and its measurement unit. Therefore, we recommend assessing various software metric tools before choosing one.

RQ3. What is the most important software quality model that can be used by the healthcare software developers to fulfil software quality requirements?

Table 4 provides information to answer this question. Our study identified the IEEE software quality model as the one with the highest number of software characteristics (i.e., 16) as shown in Figure 1. ISO/IEC 25,010 follows with 15 factors and McCall is third with 14.

8. Conclusion

Healthcare software applications play an essential role in the process of diagnosis and treatment of diseases, consequently, software malfunctions can cause threats to life or damage to the environment. Providers of health information technology consider software quality of vital importance for various reasons, one of which is to ensure maintainability of software despite changes in environments, requirements and functional specifications. In literature, different authors have studied the topic of software maintenance, giving recommendations to preserve maintainability against the growing complexity of code.

In this paper, we have presented healthcare software criticalities and software quality models, and we have investigated their relationship. Based on the aim of this study, we have provided a complete discussion on the research questions. It is out of the scope of this paper to suggest how to improve existing healthcare software, because it is under the responsibility of the software vendors that have also access necessary data (e.g., to validate and verify their software). As regards criticalities, we have described the most popular ones in healthcare software system (see Table 1) based on existing literature, software quality models and programming language standard. Concerning quality, we have detailed all the most widespread quality models, focusing our attention on their principal characteristics, as shown in Table 2, and we have derived a common terminology, as described in Table 3. Afterwards, we have compared the software quality models in terms of the healthcare software criticalities they manage to cover, the outcome is shown in Table 4 and Figure 1. Finally, we have matched healthcare software criticalities with software characteristics as presented in Table 5, Figure 2 and Table 6.

Our main contributions have been: contextualising the criticalities to the healthcare software domain (see Table 1), highlighting the subset of software characteristics (see Table 5) that should be important to control and monitor with respect to healthcare software criticalities, ranking healthcare software criticalities with respect to software characteristics (see Table 6), and defining a non-specific domain methodology. Table 5 summarises one of the major outcomes of this research by highlighting that maintainability is the software characteristic that covers more criticalities: this has been achieved by combining information of healthcare criticalities and software characteristics. Concerning Table 6, we have concluded that C2 and C3 are the criticalities that require particular attention, because they affect the highest number of characteristics: C2 is about the risk of changing hard-coded software; C3 focuses on the importance of developing maintainable software.

In literature, scientists (Aghazadeh et al., 2015; Smith et al., 2005) have examined the relationship between software quality characteristics and some quality indicators, such as user satisfaction, quality of patient care and clinical workflow and efficiency. To the best of our knowledge, no study has so far endeavoured to investigate the effects of all software quality characteristics on healthcare software. In our view, this analysis might lead to an improvement in the assessment and treatment of diseases by enhancing software quality and, therefore, reducing the risks faced by patients. This work has an ambitious goal because its applicability requires a strict collaboration between software healthcare companies and researchers in software engineering.

Acknowledgments

This research was supported by INFN.

Notes

1.

Rational Software, www.rational.com.

Disclosure statement

No potential conflict of interest was reported by the authors.

References

  1. Aghazadeh, S., Pirnejad, H., Aliev, A., & Moradkhani, A. (2015). Evaluating the effect of software quality characteristics on health care quality indicators. Journal of Health Management & Informatics, 2(3), 67–73. [Google Scholar]
  2. Aguiar, A., Filho, S. J., Magalhaes, S. J., Casagrande, T. D., & Hessel, F. (2010). Helfire: A design framework for critical embedded systems’ applications. Proceeding of the ISQED’10, 730–737. [Google Scholar]
  3. Ali, Z., Talha, M., & Alsulaiman, M. (2017, May). A practical approach: Design and implementation of a healthcare software for screening of dysphonic patients. IEEE Access, 5, 5844–5857. 10.1109/ACCESS.2017.2693282 [DOI] [Google Scholar]
  4. Al-Qutaish, R. E. (2010). Quality models in software engineering literature: An analytical and comparative study. Journal of American Science, 6(3), 166–175. [Google Scholar]
  5. Ashton, D. (2006). A practical implementation of a two level archetype based clinical model. In Proceedings of the HIC 2006 and HINZ 2006 (pp. 66–82). Informit. [Google Scholar]
  6. ASQ . (2007). American society for quality: Glossary. www.asq.org/glossary/q.html
  7. Atalag, K., Yang, H. Y., Tempero, E., & Warren, J. R. (2014). Evaluation of software maintainability with openEHR – a comparison of architectures. International Journal of Medical Informatics, 83(11), 849–859. 10.1016/j.ijmedinf.2014.07.006 [DOI] [PubMed] [Google Scholar]
  8. Awan, S., Malik, F., & Javed, A. (2015). An efficient and objective generalized comparison technique for software quality models. International Journal of Modern Education and Computer Science, 7(12), 57–61. 10.5815/ijmecs.2015.12.08 [DOI] [Google Scholar]
  9. Basil, V. R., & Turner, A. J. (1975). Interactive enhancement: A practical technique for software development. IEEE Transactions on Software Engineering, SE-1(4), 390–396. 10.1109/TSE.1975.6312870 [DOI] [Google Scholar]
  10. Beale, T. (2005). The health record: Why is it so hard? In Haux R., Kulikowski C., & Stuttgart S. (Eds.), Imia yearbook of medical informatics, ubiquitous health care systems (pp. 301–304). IMIA. [PubMed] [Google Scholar]
  11. Bengtsson, P., & Bosch, J. (1999, March). Architecture level prediction of software maintenance. In Proceedings of third European Conference on Software Maintenance and Reengineering (pp. 139–147). IEEE. [Google Scholar]
  12. Berander, P., Damm, L. -O., Eriksson, J., Gorschek, T., Henningsson, K., & Jönsson, P., … Tomaszewski, P. (2005). Software quality attributes and trade-offs. https://www.uio.no/studier/emner/matnat/ifi/nedlagte-emner/INF5180/v10/undervisningsmateriale/reading-materials/p10/Software_quality_attributes.pdf
  13. Boehm, B. W., Brown, J. R., Kaspar, H., Lipow, M., McLeod, G., & Merritt, M. (1978). Characteristics of Software Quality (TRW series of software technology). American Elsevier. [Google Scholar]
  14. Boehm, B. W., Brown, J. R., & Lipow, M. (1976). Quantitative evaluation of software quality. In Proceedings of the 2nd International Conference on Software Engineering. ACM. [Google Scholar]
  15. Dromey, R. G. (1995). A model for software product quality. IEEE Transactions on Software Engineering, 21(2), 146–163. 10.1109/32.345830 [DOI] [Google Scholar]
  16. Dromey, R. G. (1996). Concerning the chimera [software quality]. IEEE Software, 13(1), 33–43. 10.1109/52.476284 [DOI] [Google Scholar]
  17. Erbert, C., & Liedtke, T. (1995, October). An integrated approach for criticality prediction. In Sixth International Symposium on Software Reliability Engineering. IEEE. 10.1109/ISSRE.1995.497639 [DOI] [Google Scholar]
  18. FDA . (2002). General principles of software validation, final guidance for industry and fda staff. FDA. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/general-principles-software-validation [Google Scholar]
  19. Fenton, N., & Pfleeger, S. L. (1996). Software metrics - a rigorous and practical approach (2 ed.). International Thomson Computer Press. [Google Scholar]
  20. Garcés, L., Ampatzoglou, A., Avgeriou, P., & Nakagawa, E. Y. (2017). Quality attributes and quality models for ambient assisted living software systems: A systematic mapping. Information and Software Technology, 82, 121–138. 10.1016/j.infsof.2016.10.005 [DOI] [Google Scholar]
  21. Grady, R. B. (1992). Practical software metrics for project management and process improvement. Practice Hall. [Google Scholar]
  22. Hemmati, A., Carlson, C., Nayebi, M., Ruhe, G., & Saunders, C. (2017). Analysis of software service usage in healthcare communication services, In Proceedings of the IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), Prague, pp. 565–566, 10.1109/QRS-C.2017.95. [DOI] [Google Scholar]
  23. IEEE . (1990). IEEE std 610.12 - Standard glossary of software engineering terminology. IEEE. [Google Scholar]
  24. IEEE . (1996). IEEE std 12207-1996 - standard for information technology - software life cycle processes. IEEE. [Google Scholar]
  25. IEEE . (2008). IEEE std 12207-2008 – ISO/IEC/IEEE international standard – systems and software engineering – software life cycle processes. IEEE. [Google Scholar]
  26. IEEE . (2011). IEEE std 15289-2011 – ISO/IEC/IEEE systems and software engineering – Content of life cycle information products (documentation). IEEE. [Google Scholar]
  27. IEEE . (2014). IEEE std 730-2014 – IEEE standard for software quality assurance process. IEEE. [Google Scholar]
  28. ISO . (2001). ISO 9126.1:2001, software engineering - product quality, part 1: quality model. ISO. [Google Scholar]
  29. ISO . (2008). ISO/IEC CD 25010:2008, software engineering: Software product quality requirements and evaluation (SQuaRE) quality model and guide. ISO. [Google Scholar]
  30. Jacobson, I., Booch, G., & Rumbaugh, J. (1999). The unified software development process. Addison Wesley Longman, Inc. [Google Scholar]
  31. Kazman, R., Bianco, P., Ivers, J., & Klein, J. (2020a). Integrability. CMU/SEI-2020-TR-001. Software Engineering Institute, Carnegie Mellon University. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=637375 [Google Scholar]
  32. Kazman, R., Bianco, P., Ivers, J., & Klein, J. (2020b). Maintainability. Software Engineering Institute, Carnegie Mellon University, Technical Report CMU/SEI-2020-TR-006, 2020. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=650480 [Google Scholar]
  33. Khatri, S., Chhillar, R. S., & Chhikara, A. (2012). Analyzing the impact of software reliability growth models on object oriented systems during testing. International Journal of Enterprise Computing and Business Systems, 2(1). [Google Scholar]
  34. Kitchenham, B., & Pfleeger, S. L. (1996). Software quality: The elusive target. IEEE Software, 13(1), 12–21. 10.1109/52.476281 [DOI] [Google Scholar]
  35. Krogstie, J. (2006). Work distribution, methodology, and technology for ISD and maintenance. In Avison D., Elliot S., Krogstie J., & Pries-Heje J. (Eds.), The past and future of information systems: 1976-2006 and beyond (pp. 131–142). Springer. [Google Scholar]
  36. Kruchten, P. (2000). The rational unified process: an introduction (Second ed.). Addison Wesley Longman, Inc. Rational Software Inc. [Google Scholar]
  37. Lee, I., Pappas, G., Cleveland, R., Hatcliff, J., Krogh, B., Lee, P., Rubin, H., & Sha, L. (2006, April). High-confidence medical device software and systems. Computer, 39(4), 33–38. 10.1109/MC.2006.180 [DOI] [Google Scholar]
  38. Marciniak, J. J. (2002). Encyclopedia of software engineering (Vol. 2, Second ed.). Wiley. [Google Scholar]
  39. McCall, J. A., Richards, 3. P. K., & Walters, G. F. (1977). Factors in software quality - concept and definitions of software quality (p. 13441). Griffiss Air Force Base. [Google Scholar]
  40. MISRA-C . (2004). Guidelines for the use of the C language in critical systems. Motor Industry Research Association. [Google Scholar]
  41. MISRA-C++ . (2008). Guidelines for the use of the C++ language in critical systems. MISRA. [Google Scholar]
  42. Murphy, C., Raunak, M. S., King, A., Chen, S., Imbriano, C., Kaiser, G., and Osterweil, L. (2011). On effective testing of health care simulation software. In Proceedings of the 3rd Workshop on Software Engineering in Health Care (pp. 40–47). Penn Liraries. [Google Scholar]
  43. Orviz Fernández, P., David, M., Duma, D. C., Ronchieri, E., Gomes, J., & Salomoni, D. (2020). Software quality assurance in INDIGO-DataCloud project: A converging evolution of software engineering practices to support European research e-infrastructures. Journal of Grid Computing, 18(1), 81–98. 10.1007/s10723-020-09509-z [DOI] [Google Scholar]
  44. Paech, B., & Wetter, T. (2008). Rational quality requirements for medical software. In Proceedings of the 30th International Conference on Software Engineering (pp. 633–638). ACM. [Google Scholar]
  45. Pau, K. (2020). Framework for reusable test case generation in software system testing. 10.4018/978-1-7998-2531-9.ch009 [DOI] [Google Scholar]
  46. Pigoski, T. M. (1996). Practical software maintenance: Best practices for managing your software investment. Wiley. [Google Scholar]
  47. Plsek, P. E., & Greenhalgh, T. (2001). Complexity science: The challenge of complexity in health care. BMJ, 323(7313), 625–628. 10.1136/bmj.323.7313.625 [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Pronovost, P. J., Goeschel, C. A., Olsen, K. L., Pham, J. C., Miller, M. R., Berenholtz, S. M., Bryan Sexton, J., Marsteller, J. A., Morlock, L. L., Wu, A. W., Loeb, J. M., & Clancy, C. M. (2009). Reducing health care hazards: Lessons from the commercial aviation safety team. Health Affairs, 28(3), 479–489. 10.1377/hlthaff.28.3.w479 [DOI] [PubMed] [Google Scholar]
  49. Rector, A. L. (1999). Clinical terminology: Why is it so hard? Methods of Information in Medicine, 38(4–5), 239–252. 10.1055/s-0038-1634418 [DOI] [PubMed] [Google Scholar]
  50. Richardson, I., Reid, L., & O’Leary, P. (2016). Healthcare systems quality: Development and use. In Proceedings of the IEEE/ACM International Workshop on Software Engineering in Healthcare Systems (SEHS), Austin, TX, pp. 50–53, 10.1109/SEHS.2016.017. [DOI] [Google Scholar]
  51. Ronchieri, E., & Pia, M. G. (2018). Assessing software quality in high energy and nuclear physics: The geant4 and ROOT case studies and beyond. In Proceedings of 2018 IEEE Nuclear Science Symposium and Medical Imaging Conference. IEEE Xplore. [Google Scholar]
  52. Samadhiya, D., Chen, D., & Wang, S. -H. (2010). Quality models: Role and value in software engineering. In Proceedings of the International Conference on Software Technology and Engineering (ICSTE). IEEE Xplore. [Google Scholar]
  53. Sharma, C., & Dubey, S. K. (2015). A perspective approach of software reliability models and techniques. ARPN Journal of Engineering and Applied Sciences, 10(16). [Google Scholar]
  54. Smith, K., Smith, V., Krugman, M., & Oman, K. (2005). Evaluating the impact of computerized clinical documentation. Computers, Informatics, Nursing : CIN, 23(3), 132–138. 10.1097/00024665-200505000-00008 [DOI] [PubMed] [Google Scholar]
  55. Sommerville, I. (2000). Software engineering (Sixth ed.). Addison Wesley. [Google Scholar]
  56. Son, H. S., & Koo, S. R. (2009). Software reliability improvement techniques. In Seong P. H. (Ed.), Reliability and risk issues in large scale safety-critical digital control systems (pp. 105–120). Springer. [Google Scholar]
  57. Stahl, T., Voelter, M., & Czarnecki, K. (2006). Model-driven software development: Technology, engineering, management”. John Wiley & Sons. [Google Scholar]
  58. Wagner, S. (2013). Software product quality control. Springer-Verlag. [Google Scholar]
  59. Weber-Jahnke, J. H., Price, M., & Williams, J. (2013, May). Software engineering in health care. Is it really different? and how to gain impact. In 5th International Workshop on Software Engineering in Health Care (SEHC). 10.1109/SEHC.2013.6602469 [DOI] [Google Scholar]
  60. Zeiss, B., Vega, D., Schieferdecker, I., Neukirchen, H., & Grabowski, J. (2007). Applying the ISO 9126 quality model to test specifications - exemplified for TTCN-3 test specifications. In W.-G. Bleek, J. Raasch, & H. Züllighoven (Eds.), Software Engineering 2007–Fachtagung des GI-Fachbereichs Softwaretechnik (pp. 231–242). Bonn: Gesellschaft für Informatik e. V. [Google Scholar]

Articles from Health Systems are provided here courtesy of Taylor & Francis

RESOURCES