Skip to main content
NCBI home page
Digital Health logoLink to Digital Health
. 2022 Jun 16;8:20552076221109055. doi: 10.1177/20552076221109055

The quality of digital health software: Should we be concerned?

Peter Kokol 1,2,✉, Helena Blažun Vošner 3,4,5, Marko Kokol 1,6, Jernej Završnik 3,4
PMCID: PMC9210082  PMID: 35746952

Abstract

The digitalization of healthcare fuelled by advances in technology and the increased prevalence of mobile smart devices and health-related internet of things can offer equitable access to expert-level healthcare globally. Growing demand for telemedicine, mobile health apps, and advanced data analytics have further established their role in a modern information society during the Covid-19 crisis. Digital health is, in essence, powered by software (DHSW), which has to operate in the specific digital health environment characteristics and is therefore highly and intrinsically complex and prone to software defects and faults. Given the lack of standardization regarding DHSW quality, we explored the available reviewed research on this crucial topic in this brief paper, using a synthetic thematic analysis approach. We assert that neither the volume, distribution nor scope of the DHSW quality research content is satisfactory, and significant research gaps exist. Based on the presented evidence, we can only conclude that we should be concerned and that the time to act is now to ensure that the unavoidable increase of usage and prevalence of DHSW will not – in the end – reduce the quality of care due to subpar software and software-based digital health systems.

Keywords: digital health, technology, eHealth, health informatics, mixed methods, studies

Introduction

Digital health technologies are becoming ubiquitous, and if successfully implemented and delivered, can offer equitable access to expert-level healthcare on a global level, thus reducing the global health and quality of life gap.1–3. The digitalization of healthcare is being fuelled by further advances in technology and the increased prevalence of mobile smart devices. It has also flourished during the Covid-19 pandemic, with the growing demand for telemedicine, employment of mobile health apps, and advanced data analytics to improve health systems.4,5 Digital health is, in essence, powered by software (DHSW), which has to operate in the specific digital health environment characteristics and therefore has to deal with several complex requirements. Among those are management of multimodal and high dimensional digital health data, 6 patient safety, 7 cyber security, 8 communication with a myriad of healthcare internet of thing devices, 9 and work in embedded medical devices in real-time.10,11

The above responsibilities make DHSW extremely and intrinsically complex, resulting in increased incidence and proneness to software defects and faults. 12 However, the number of faults is not the only important quality metric for DHSW; the perceived quality of the DHSW by stakeholders, patients, health professionals, and other end-users or software developers is also paramount. Therefore, various standards and software quality models define different software quality characteristics: the available functionality, reliability, usability, portability, or maintainability, which should all be present in DHSW. 13

Surprisingly, there is only one international standard regarding the regulation of DHSW, namely the International Electrotechnical Commission (IEC) 62304, 14 which was also accepted by the U.S. Food and Drug Administration (FDA). However, FDA additionally requires the use of rigorous validation approaches. 15 Nevertheless, both FDA and IEC provide just a very general description of common DHSW development life cycle activities, without any strict indication concerning process models or methods and techniques for DHSW quality assurance or development. 16

Given the lack of standardization, we explored and reviewed the DHSW quality using a synthetic thematic analysis approach. Within that framework, the volume and trends of the research literature production, its spatial distribution and content are analysed, and we address potential research gaps and propose possible solutions.

Methodology

Synthetic thematic analysis is a triangulation of bibliometric mapping and thematic analysis. 17 It enables one to holistically assess the research activity in any scientific field by semi-automatically analysing the corpora of scientific publications. Unlike traditional and more formal knowledge synthesis methods, which are labour intensive and limited to a relatively small number of publications, it enables one to process several hundreds or even thousands of publications.

We first harvested the publications from the Scopus database and then performed the authors’ keywords based on bibliometric mapping of extracted publications using VOSViewer software. 18 Scopus was selected due to the fact that it is the largest bibliographic database of reviewed literature. To find the articles the search string TITLE-ABS-KEY({quality software} OR {software quality}) AND (LIMIT-TO (SUBJAREA, ‘MEDI’) OR LIMIT-TO (SUBJAREA, ‘HEAL’) OR LIMIT-TO (SUBJAREA, ‘NURS’) OR LIMIT-TO (SUBJAREA, ‘DENT’)) was used where MEDI denotes the subject category Medicine, HEAL health, NURS nursing, and DENT dentistry. The resulting author keywords cluster landscape was then thematically analysed using authors’ keywords as codes and links between authors’ keywords to form categories. Finally, subcategories were used to label clusters with themes.

Results

Volume of research

The search resulted in 259 publications, of which there were 137 conference papers, 105 articles, 8 reviews and 11 other types of publications, such as books and conference reports. In comparison, the search for the same keywords not restricted to Scopus subject areas resulted in 13,429 publications.

Given the result, it is clear that medical software quality research represents a minuscule part (<2%) of the overall software quality research. Given the importance of the DHSW and the possibly catastrophic consequences of software failures within DHSW, the indicated lack of interest in research focusing on software is surprising and worrying. To confirm the findings, we performed additional searches using strings, such as ‘TITLE-ABS-KEY(({quality software} OR {software quality}) AND (medicine or nursing or health or dent*)) or TITLE-ABS-KEY (“digital health” AND software AND quality)’, which resulted in comparable corpora containing 94 to 240 publications.

The oldest publication indexed in Scopus dates to 1979 when Srinivasan and Dascher 19 published their paper on producing and managing health care software. During the first period (1979–2010), publications were scarce, fluctuating between zero to seven per year. The notable increase in productivity occurred in 2010 when exponential growth began, reaching the peak productivity in 2019 with 40 publications. After that, the number of publications dropped to 28 in 2020. The year 2021 is not finished, but the current number of 16 publications might indicate that the negative production trend will continue.

Spatial distribution of research

The literature production was spread between 54 countries. The most productive country was the United States (n = 55), followed by India (n = 24), Brazil (n = 22), Germany (n = 17), the United Kingdom (n = 17), China (n = 13), Canada (n = 11), and Italy (n = 10). Those countries produced more than two-thirds of all publications, so a regional concentration of research in G8 and other developed countries with good economies and health systems can be observed.

Among 103 source titles, the most prolific sources were Studies in Health Technology and Informatics (n = 16), Proceedings IEEE Symposium On Computer-Based Medical Systems (n = 12), Medical Physics (n = 7), and International Journal of Medical Informatics (n = 5). It is interesting to note first that the most prolific source titles are conference proceedings. Second, the rest of the publication production is spread among more than 90 journals, meaning that no core archival journals have been established yet, indicating that research is still reaching maturity.

Research themes

The authors’ keyword landscape is shown in Figure 1. Among 742 author keywords, 77 occurred in more than one publication. Bibliometric mapping distributed those 77 keywords into six clusters. Selecting more popular keywords as codes and analysing links between them resulted in themes presented in Table 1.

Figure 1.

Figure 1.

Open in a new tab

The cluster landscape of author keywords emerging in more than one paper. (VOSViewer resolution = 0.25).

Table 1.

DHSW quality research themes (numbers in parenthesis represent the number of publications in which an author keyword occurred three or more times).

Theme Popular codes Categories
User satisfaction (10 keywords) User satisfaction (3); sentiment analysis (3); software process (2); certification (2); image-guided therapy (2) Sentiment analysis in user satisfaction assessment; certification of software processes; software platforms for image-guided therapy
Machine learning in empirical software engineering (15 keywords) Software engineering (8); machine learning (8); software metrics (5); software testing (4); software defect prediction (4); refactoring (4) Software engineering supported by machine learning and data mining of software metrics data; machine learning for software defect prediction based on software quality metrics; data mining in software testing; software maintenance based on refactoring, design patterns and software metrics
Improving patient safety with standardized health information systems (9 keywords) Patient safety (6); electronic health records (4); clinical decision support systems (4); health information systems (3) Test-driven software development of clinical decision support systems based on electronic health records to improve patient safety, standards for health information systems and electronic health systems
Software quality assurance of DHSW focusing on reliability (26 keywords) Software (11); quality assurance (8); open source software (7); software quality assurance (6); information systems (5); evaluation (5) CMMI guidelines for software development evaluation; quality assurance guidelines for radiation therapy; software quality assurance for information system development; software reliability of open source software
Quality evaluation of telemedicine software (5 keywords) Case study (6); software quality evaluation (5); telemedicine (3) Case studies in software quality evaluation of telemedicine applications
ISO/IEC software quality standards in mobile applications (12 keywords) Software quality (39); ISO/IEC 25010 (5); software evaluation (3); pregnancy (3); data quality (3) ISO/IEC standards in software quality related to mobile applications and personal health records; ISO/IEC standards in mobile personal health records related to pregnancy; software quality and data quality
Open in a new tab

CMMI: Capability Maturity Model Integration; IEC: International Electrotechnical Commission.

Research gaps

The above analysis revealed that DHSW quality research focuses on two quality features: reliability and user satisfaction. The software quality research is mainly concerned with mobile applications, telemedicine, radiation and image therapies, and clinical decision systems regarding digital health domains. From the quality point of view, quality assurance is performed by following ISO/IEE standards, capability maturity model integration, and using machine learning in empirical software engineering practices, such as software defect prediction.

Comparing DHSW quality research to the overall software quality research 20 following potential significant research gaps can be identified:

  • - Using knowledge management in DHSW development.

  • - Consideration of human factors and software complexity in DHSW evaluation.

  • - Including project management aspects in DHSW development life cycles.

  • - Analysing if the employment of agile approaches increases the quality of DHSW.

  • - How to manage/assure the quality of embedded DHSW.

  • - How to improve the DHSW quality with software reuse.

Discussion and conclusion: Should we be concerned?

According to our synthetic review, neither the volume, distribution nor scope of the DHSW quality research content is satisfactory, especially given the ever-growing importance of DHSW and the potential consequences of quality defects. In our opinion, a systematic approach to DHSW quality is required, and this can only follow from a robust, well-defined and well-researched body of evidence. Based on identified existing trends and best practices in overall software quality, substantial research in software quality exists but would need to be tailored to the specific requirements of DHSW quality.

Should we, therefore, be worried based on the current trends and state-of-the-art? We believe so. Nevertheless, concrete actions can (and should) be taken by the research and broader professional community to ensure that the unavoidable increase in usage and prevalence of DHSW will not – in the end – reduce the quality of care due to subpar software and software-based tools.

Possible next steps could include several approaches. An interdisciplinary review of software quality methods already utilized and identifying the necessary adaptations to the DHSW could be performed. Additionally, evaluating the usefulness of methodologies researched in other critical industries (such as aviation) would probably be beneficial. Existing industry best practices that (probably) exist should be systematically researched, as it seems they did not receive adequate independent vetting and analysis required to ensure their usefulness and robustness. International Medical Informatics Association Technology Assessment & Quality Development in Health Informatics Working Group, the newly established European Federation of Medical Informatics Working Group on Digital Health and similar associations could be informed about the problem and urged to organize workshops, special conference sessions, and special journal issues concerning DHSW quality. Finally, on the political level, research funding agencies could be informed and asked to prepare necessary policies to ensure more funding for DHSW quality research.

Footnotes

Contributorship: All four authors have equally contributed to the conceptualization of the study and the paper. PK performed the data curation, and PK and MK performed formal analysis. All four authors have contributed to the development of the methodology and validation of the study. PK and MK have written the original draft, and HB and JZ performed work on reviewing paper revisions, and editing.

Declaration of conflicting interests: The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval: Not applicable, because this article does not contain any studies with human or animal subjects.

Funding: The authors received no financial support for the research, authorship, and/or publication of this article.

Guarantor: PK.

References

  • 1.Ibrahim H, Liu X, Zariffa N, et al. Health data poverty: An assailable barrier to equitable digital health care. Lancet Digit Health 2021; 3: e260–e265. [DOI] [PubMed] [Google Scholar]
  • 2.WHO Global Strategy on Digital Health 2020–2025. The Compass for SBC, 2019. https://www.thecompassforsbc.org/sbcc-tools/who-global-strategy-digital-health-2020-2025 (accessed 16 November 2021).
  • 3.Perakslis E, Ginsburg GS. Digital health—the need to assess benefits, risks, and value. JAMA 2021; 325: 127–128. [DOI] [PubMed] [Google Scholar]
  • 4.Figueroa CA, Luo T, Aguilera A, et al. The need for feminist intersectionality in digital health. Lancet Digit Health 2021; 3: e526–e533. [DOI] [PubMed] [Google Scholar]
  • 5.Gordon WJ, Landman A, Zhang H, et al. Beyond validation: Getting health apps into clinical practice. NPJ Digit Med 3. 14. DOI: 10.1038/s41746-019-0212-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Berisha V, Krantsevich C, Hahn PR, et al. Digital medicine and the curse of dimensionality. NPJ Digit Med 2021; 4: 1–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Sujan M, Scott P, Cresswell K. Digital health and patient safety: Technology is not a magic wand. Health Inf J 2020; 26: 2295–2299. [DOI] [PubMed] [Google Scholar]
  • 8.Giansanti D. Cybersecurity and the digital-health: The challenge of this millennium. Healthcare 2021; 9: 62. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Habibzadeh H, Dinesh K, Rajabi Shishvan O, et al. A survey of healthcare internet of things (HIoT): A clinical perspective. IEEE Internet Things J 2020; 7: 53–71. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Kadakia K, Patel B, Shah A. Advancing digital health: FDA innovation during COVID-19. NPJ Digit Med 2020; 3: 1–3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Gordon WJ, Stern AD. Challenges and opportunities in software-driven medical devices. Nat Biomed Eng 2019; 3: 493–497. [DOI] [PubMed] [Google Scholar]
  • 12.Sheketa V, Zorin V, Chupakhina S, et al. Empirical method of evaluating the numerical values of metrics in the process of medical software quality determination. In: 2020 international conference on decision aid sciences and application (DASA). New Jersey: IEEE, 8–9 November 2020, pp.22–26. [Google Scholar]
  • 13.Chen C, Shoga M, Boehm B. Exploring the dependency relationships between software qualities. In: 2019 IEEE 19th international conference on software quality, reliability and security companion (QRS-C). New Jersey: IEEE, 22–26 July 2019, pp.105–108. [Google Scholar]
  • 14.Jordan P. Standard IEC 62304 – medical device software – software lifecycle processes. 2006; 41–47.
  • 15.Health C for D and R. General principles of software validation. U.S. Food and Drug Administration, FDA-1997-D-0029, 2019, https://www.fda.gov/regulatory-information/search-fda-guidance-documents/general-principles-software-validation (2020, accessed 9 November 2021). [Google Scholar]
  • 16.Arcaini P, Bonfanti S, Gargantini A, et al. Integrating formal methods into medical software development: The ASM approach. Sci Comput Program 2018; 158: 148–167. [Google Scholar]
  • 17.Kokol P, Kokol M, Zagoranski S. Code smells: A synthetic narrative review. Lib Philos Pract (e-Journal) 2020; 2020. https://digitalcommons.unl.edu/libphilprac/3507 (accessed 12 November 2012) [Google Scholar]
  • 18.van Eck NJ, Waltman L. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics 2010; 84: 523–538. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Srinivasan CA, Dascher PE. Manage each step for quality software. Hosp Financ Manage 1979; 33: 18–24. [PubMed] [Google Scholar]
  • 20.Kokol P. Software quality: A historical and synthetic content analysis. arXiv:210614598 [cs], http://arxiv.org/abs/2106.14598 (2021, accessed 24 November 2021).

Articles from Digital Health are provided here courtesy of SAGE Publications

RESOURCES