Digital health and capability maturity models—a critical thematic review and conceptual synthesis of the literature

Abstract

Objective

A literature review of capability maturity models (MMs) to inform the conceptualization, development, implementation, evaluation, and mainstreaming of MMs in digital health (DH).

Methods

Electronic databases were searched using “digital health,” “maturity models,” and related terms based on the Digital Health Profile and Maturity Assessment Toolkit Maturity Model (DHPMAT-MM). Covidence was used to screen, identify, capture, and achieve consensus on data extracted by the authors. Descriptive statistics were generated. A thematic analysis and conceptual synthesis were conducted.

Findings

Diverse domain-specific MMs and model development, implementation, and evaluation methods were found. The spread and pattern of different MMs verified the essential DH foundations and five maturity stages of the DHPMAT-MM. An unanticipated finding was the existence of a new category of community-facing MMs. Common characteristics included:

1. A dynamic lifecycle approach to digital capability maturity, which is:

a. responsive to environmental changes and may improve or worsen over time;

b. accumulative, incorporating the attributes of the preceding stage; and

c. sequential, where no maturity stage must be skipped.

2. Sociotechnical quality improvement of the DH ecosystem and MM, which includes:

a. investing in the organization’s human, hardware, and software resources and

b. a need to engage and improve the DH competencies of citizens.

Conclusions

The diversity in MMs and variability in methods and content can create cognitive dissonance. A metamodel like the DHPMAT-MM can logically unify the many domain-specific MMs and guide the overall implementation and evaluation of DH ecosystems and MMs over the maturity lifecycle.

INTRODUCTION

The WHO Global Strategy on Digital Health 2020–2024¹ defines digital health (DH) as “the field of knowledge and practice associated with any aspect of adopting digital technologies to improve health, from inception to operation” and specifies a key output as “a dynamic digital health maturity model assessment to guide prioritization of national investment in digital health to support primary health care and universal health coverage.” Given the iterative and quality improvement emphasis inherent in the health technology lifecycle, a capability maturity approach is useful to guide the development, adoption, adaptation, implementation, and maintenance of DH.^2,3

The World Health Organization (WHO) Resolution on Digital Health urges countries to assess their digital health maturity (DHM) to identify and prioritize areas that may benefit from digital technologies as “a means of promoting equitable, affordable and universal access to health for all, including the special needs of groups that are vulnerable”⁴ within the context of the relevant United Nations’ Sustainable Development Goals.⁵

“Capability” is the physical and/or mental ability to develop, improve and actualize skills and competencies, while “maturity” describes a stage of growth or development. Therefore, “capability maturity” of an organization refers to the stage of growth of capability. A maturity model (MM) describes the growth and development of capability maturity over time in discrete but coordinated stages which describe the organizational behaviors, practices, and processes that reliably and sustainably produce the required outcomes.⁶ Monitoring and evaluation of organizational behaviors, operating procedures, and policies along the maturity stages is quality measurement and improvement.

The Digital Health Profile and Maturity Assessment Toolkit

A capability maturity approach was used to develop the Digital Health Profile and Maturity Assessment Toolkit (DHPMAT).⁷ With its origins in the World Health Organisation (WHO) and International Telecommunication Union (ITU) National eHealth Strategy Toolkit,⁸ the DHPMAT Maturity Model (DHPMAT-MM) describes five essential DH foundations: information communication technology infrastructure, access, and equity; essential DH tools; readiness to share information; enablers of trust and adoption; and quality improvement, monitoring, and evaluation (QIME) and five stages of maturity: basic, controlled, standardized, optimizing and, innovating (Figure 1).⁷ The stage of maturity is determined by an assessment of the system performance and extent of knowledge sharing at micro, meso, and macro levels of the organization and enterprise. Assessment uses qualitative and/or quantitative information from a range of published and unpublished internal sources. Information will also be sought from relevant key informants to add to or verify collected information.

Figure 1.

The DHPMAT-MM used in the literature review. DHPMAT-MM: Digital Health Profile and Maturity Assessment Toolkit Maturity Model.

Figure 1 illustrates the relationships within the framework and criteria used to stage the maturity of each foundation. At one end, the infrastructure is assessed for stability, reliability, coverage, access, and affordability; at the other, the assessment of the DH strategy, investment, governance, leadership, and workforce. All foundations have a QIME plan and the maturity of QIME as a DH foundation is assessed on the extent of its coordination from local project evaluation, through organization and enterprise quality improvement, to a nationally coordinated comparative effectiveness research program.

Objective

A critical thematic review of the capability MM literature was conducted to inform the conceptualization, development, implementation, and evaluation of Digital Health Maturity Models (DHMMs), using the DHPMAT-MM as the starting point.⁷

The review was constructed to address the following questions:

1. What have capability MMs been used for—why and what for—in DH?

2. How have DHMMs been conceptualized and developed?

3. How have DHMMs been implemented and evaluated (up to and including field-tests)?

4. What has been the “spread and scale-up”⁹ of DHMMs?

METHODS

The design and conduct of the review were guided by and complied with the PRISMA requirements. COVIDENCE was used to implement the review. The DHPMAT-MM⁷ (Figure 1) and the research questions guided the search strategy (see Supplementary File), title and abstract review, full-text appraisal, data extraction, and synthesis. Data were extracted to describe the settings, essential DH foundations, stages, and attributes of capability maturity, and how the MMs were developed, implemented, and evaluated. We included reviews, applied research, and conceptual papers by recognized experts on digital maturity frameworks. No time limit was set.

The inclusion criteria included: the model rationale, purpose, and development are described along with any underlying theory and setting.

The search strategy was developed using the key words “digital health” and “maturity models” (and their related terms) with Boolean operators. The bibliographic databases were PubMed, Embase, and Scopus. The search strategies and the number of records identified from searches are presented in the Supplementary Material.

Data extraction was done independently, using Covidence, to collect information on: capability MM purpose/use; model custodian; literature review for other models; model conceptualization, development, implementation, and evaluation (up to and including field-tests), and the extent of “spread and scale-up” of the model. We defined “spread” as replicating an initiative somewhere else and “scale-up” as tackling the infrastructural problems (across an organization, locality, or health system) that arise during full-scale implementation. However, in practice, one blurs into the other.^9,10

Consensus on the themes identified in the extracted data was achieved through iterative discussions using the DHPMAT-MM as the starting point. A random subset of articles were recoded to ensure that the data extraction, themes identification, and consensus process were consistent and the outcomes reproducible. The COVIDENCE dataset was transferred to an Excel spreadsheet for further data cleaning and analyses.

FINDINGS

The papers identified, screened, and included are summarized in a PRISMA diagram (Figure 2). There were 765 papers reported from 750 studies. Application of the inclusion criteria derived from the conceptual framework and research questions resulted in 65 papers for data extraction.

Figure 2.

PRISMA diagram of papers screened and included.

The diversity of domains, purposes, uses, and users of MMs

MMs have proliferated across many disciplines, domains, settings, and organizations as tools for describing a specific development status, circumstance, or condition of an organization, a process or a structure.¹¹ The disciplines were health, computer science and engineering, and management and social sciences. The identified topics were related to diverse areas such as clinical disciplines,^12,13 managerial^14,15 and operational matters,^16–18 quality improvement,^19,20 knowledge management,^21–23 data analytics,^24,25 policy,²⁶ governance,^16,27–29 or particular DH constructs such as social media,^30,31 software processes, cybersecurity,³² or standards for digital data quality³³ and interoperability.^34,35

The settings were mostly hospital-based and secondary care, with relatively few on primary care or community.

Table 1 summarizes the range of general and DH-specific MMs found and their key components were re-organized within the DHPMAT-MM of five essential foundations. An additional category which did not fit into the DHPMAT-MM was identified: community-facing and patient-centeredness MMs. A full list is presented in the Supplementary Material.

Table 1.

Identified domain-specific MMs (rows) mapped to the DHPMAT-MM features (columns)

Discipline		Five essential DH foundations
		1. ICT and IoT infrastructure		2. Essential digital tools				3. Readiness for information sharing					4. Enablers of trust and adoption					5. QIME
		Telecoms infrastructure, supply chain, etc	Network/Internet access, penetration, affordability, reliability, etc.	Unique Identification system	Health information system/eHR/eMR	mobileHealth, teleHealth	Health reporting	Social media	Standards and interoperability	Data quality	Hardware, software and protocols	Security and privacy	Leadership and Governance	Management and Organization	Policy and Regulation	Workforce and Capacity building	Strategy and Investment	Quality improvement, monitoring and evaluation
Management and Social sciences	Brooks et al36						X			X	X	X	X	X	X	X	X	X
	Comuzzi and Patel25									X	X	X	X	X
	Domlyn et al37												X	X	X			X
	Dos Santos and De Fátima Marin38													X				X
	Ekionea and Fillion21	X	X				X				X		X	X	X	X	X
	Gomes et al39												X	X	X	X	X	X
	Goncalves Filho and Waterson16												X	X	X	X
	Handfield40													X
	Kouroubali et al41			X	X				X			X	X	X		X		X
	Krasuska et al42			X	X	X	X		X	X	X	X	X	X		X	X	X
	Krey28												X	X
	Larsson43						X		X		X	X		X	X
	Liebe and Hübner20				X		X		X	X	X			X
	Lima et al15													X
	McCarthy et al29												X	X	X	X
	Poghosyan et al44	X	X		X						X		X	X	X	X	X	X
	Proença and Borbinha45								X	X	X		X	X			X	X
	Salah et al46																	X
	Saltz 47												X	X	X	X	X	X
	Schlichter 48												X	X		X		X
	Storm et al49												X	X	X		X	X
	Thomas and Woodside50							X
	Williams et al51	X	X		X	X	X		X	X	X	X
	Yeon and Hwang52	X	X	X	X	X	X		X	X	X	X	X	X	X	X	X	X
	Younoussi and Roudies53			X	X	X	X	X							X		X
Health	Agarwal et al54				X		X			X			X	X
	Alexander et al55	X	X		X	X			X	X	X	X	X	X	X	X
	Baltaxe et al13	X	X		X	X	X	X	X	X	X	X	X	X	X		X	X
	Chong et al56				X					X				X
	Daclin et al57								X		X
	European Society of Radiology58	X	X		X		X		X		X	X					X
	Flott et al12								X							X	X	X
	Gillies59				X		X
	Gillies and Howard60	X	X	X	X		X		X		X	X	X	X		X
	Gomes and Romão2				X		X							X			X
	Hillson61							X					X	X
	Junger and Burgert62																	X
	Kenneally et al63				X		X		X		X		X	X		X
	Knosp et al23	X	X		X		X	X	X	X	X	X	X	X	X	X	X	X
	Liaw et al26	X	X		X	X	X		X	X	X	X	X	X	X	X	X	X
	Liaw et al7	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X
	Liu et al64	X	X		X	X	X		X		X			X		X
	Mettler and Pinto65	X	X	X	X	X	X		X	X	X	X	X	X	X	X	X	X
	Orenstein et al66				X		X				X		X	X	X		X	X
	Pak and Song67	X	X		X	X	X				X
	Rafiei et al68												X	X	X		X	X
	Tcheng et al69								X	X			X	X			X
	Tom-Aba et al70					X					X	X	X	X	X		X	X
Computer science and Engineering	Akinsanya et al71	X		X							X	X	X	X	X
	Athanasiou et al72		X											X		X
	Baolong et al73	X			X		X			X		X		X
	Burmann and Meister74				X				X	X		X	X	X	X
	Carvalho et al75	X	X		X		X			X		X	X	X		X	X	X
	Daraghmeh and Brown24				X						X	X	X	X
	Guedria et al34	X	X		X				X		X			X
	Jara et al32				X				X			X	X	X	X
	Blondiau et al76	X	X		X		X			X	X		X	X	X	X	X	X
	Pulparambil and Baghdadi77	X	X		X						X		X	X			X	X
	Romero-Lopez-Alberca et al78								X		X			X	X
	Stoldt et al79	X	X		X		X		X	X		X	X	X	X			X
	Vallerand 201780								X		X			X		X
	Van de Wetering and Batenburg81				X		X		X		X	X	X	X	X	X	X	X
	Van Dyk et al82	X	X			X					X		X	X	X	X	X	X
	Van Dyk83	X	X			X				X								X
	Van Velsen et al84								X		X
Community and patient	Alexander et al85												X	X	X	X	X	X
	Domlyn et al37												X	X	X			X
	Pak and Song67	X	X		X	X	X				X
	Flott et al12								X							X	X	X
	Baltaxe et al13	X	X		X	X	X	X	X	X	X	X	X	X	X		X	X
	Birch and Heffernan86							X					X	X
	Thomas and Woodside50							X

Note: Domlyn et al, Flott et al, Baltaxe et al, Pak and Song, and Thomas and Woodside recur in the section on “community-engagement and person-centered activities.”

Abbreviations: DHPMAT-MM: Digital Health Profile and Maturity Assessment Toolkit Maturity Model; ICT: information communication technology; MMs: maturity models; QIME: quality improvement, monitoring, and evaluation.

Geographical distribution and target users

Most studies were national (n = 32) or local (n = 10) in scope, with 13 international studies. The categorization of local implies one institution in 1 state/province; it also applies to early untested MM development. There were only a few community-focused or citizen-engagement papers. Many papers were from the management and social science disciplines; less were from health or computer science and engineering. Table 2 summarizes.

Table 2.

Geographical distribution and target users of capability MMs

	Count	Definitions and comments
Geographical scope of study (N = 55 because 6 had no information and 4 were review papers)
Global	7	Defined as >1 nation in >1 region involved
Regional	6	Defined as WHO regions: Europe, Africa, etc.
National	32	Defined as multiple local sites within one country
Local	10	Defined to include early model development
Target user group
Community	4	Includes individuals, families, and communities
Clinician	34	Includes all involved with direct patient care
Manager	54	Includes managerial and other admin staff
Data users	28	Includes researchers, epidemiologists, data analysts
Policymakers	41	Included policymakers and regulators
Other	13
Disciplines (note: the multidisciplinary nature of digital health means that some papers will fall into more than 1 category and totals w ill not add up)
Computer science and engineering	18	Includes: data science, systems engineering, health informatics, health information management, telemedicine, enterprise architecture
Health	22	Includes: health informatics, health management, clinical workflow, health policy, research, health information management, digital data quality, digital global goods
Management and social sciences	24	Includes: business intelligence, management engineering, community development, knowledge management, systems management, pharmaceutical marketing, public services, information management, governance, enterprise architecture, clinical workflow, business process, professional regulation, occupational safety and health, big data, project management, health policy, social media, infrastructure, digital convergence across public services, and software reuse

Abbreviation: WHO: World Health Organization.

Robustness of model development

Of the 65 papers identified, 43 (66%) reported the process of model development and 38 (58%) included a literature review as part of the process. Fifty-two (80%) used a conceptual framework which had been developed through an expert consensus. Only 10 (15%) used or based aspects of the development on standards. There was ambiguity in the use of the term “standards” as in “ISO standards” versus “interoperability standards.” Just over half (35) of the models progressed to evaluation, which mostly used observational methodologies and mixed (quantitative and qualitative) data collection methods. Table 3 summarizes.

Table 3.

Quality of reporting and types of evaluation study designs

Quality indicators	Yes	No	NA	Reported elsewhere
Model development reported	43	16	5	1
Lit review as part of development	38	20	6	1
Conceptual framework present	52	7	5	1
Standards used	10	45	10	0
Expert consensus used	35	16	5	1
Model evaluated/field tested	35	24	6	0
Methods used for evaluation
Observational study	15	Mixed data collection methods were most frequently used in comparative studies. Purely quantitative or qualitative methods were less frequently used.
Observational comparative	22
Mixed data collection	27
Qualitative data only	8
Quantitative data only	2

Coverage of the DHPMAT-MM essential DH foundations

The relatively even distribution of the papers across maturity stages 1–4 generally as well as by user groups probably reflects the pattern of DHM in the “real world” (Table 4). The lower numbers in the innovating maturity stage, which requires significant resources, is practically and clinically significant. It also reinforces this “real-world” assumption.

Table 4.

Foundations and stages of maturity addressed by included capability maturity papers

Stage of maturity (collapsed from 5 to 3 stages)	Total counta	User group
		Individual	Clinician	Manager	Data user	Policymakers
Emerging systems
1. Basic	38	3	25	38	19	29
2. Controlled	41	3	27	45	21	32
Standardizing
3. Standardized	46	3	29	49	24	36
Clusters of excellence
4. Optimizing	42	4	28	48	24	35
5. Innovating	13	3	11	13	6	13
Digital health foundation
ICT and IoT infrastructure (n = 45)
Telecoms infrastructure, etc.	23	3	6	20	10	15
Network readiness, access, etc.	22	3	16	18	9	15
Essential digital tools (n = 88)
Unique identification system	8	1	5	4	3	7
HIS/eHR/eMR	33	3	24	29	16	22
mHealth, telehealth	14	2	10	10	5	12
Health reporting	27	2	20	24	13	19
Social media	6	1	4	6	2	3
Readiness for information sharing (n = 108)
Standards and interoperability	28	4	19	24	13	21
Data quality	22	2	17	19	11	18
Hardware, software, protocol	34	3	19	29	17	25
Security and privacy	24	3	16	21	15	18
Enablers of trust and adoption (n = 182)
Leadership and governance	40	3	25	37	19	29
Management and organization	53	3	27	37	26	36
Policy and regulation	29	2	17	26	15	22
Workforce and capacity building	26	3	16	22	14	19
Citizen engagement and DH competencies	7	4	4	4	2	3
Strategy and investment	27	2	16	25	14	20
Quality improvement, monitoring, and evaluation (QIME)
QIME	30	2	17	25	15	20

Abbreviations: DH: digital health; ICT: Information communication technology.

^aUser groups are not mutually exclusive across studies, so numbers do not add up to 65.

Table 4 summarizes the distribution of the various subcategories of the essential DH foundations by target groups. The overall pattern as well as distribution by target groups are probably consistent with the current organizational interest areas in the “real world.”

The maturity of unique Identification systems is less frequently studied (12%), as is social media (9%). There is also little focus on citizen-engagement studies (11%). Most of the studies are in “enablers of trust and adoption” (40%–82%), again with little emphasis on the community. There is a good emphasis on QIME (46%). These models have been developed to guide digital implementation and evaluation for safety and quality improvement and to inform the approach and focus for improving the digital capability maturity of a specific domain or specialized area within an organization.⁸⁷

CONCEPTUAL SYNTHESIS OF THE LITERATURE REVIEW FINDINGS

Verifying and validating the DHPMAT-MM

The distribution of the MM studies (Table 4) supports the view that DHPMAT-MM DH foundations are a logical categorization that covers a comprehensive range of DH concepts and actors. This is essential to the adequate implementation and quality improvement of DH programs undertaken by health systems and organizations.

The pattern of the research topics appears to reflect current priorities and challenges in the evolving real-world DH enterprise. Unique ID systems are less frequently studied, partly explaining why interoperability and enterprise planning remain largely aspirational. Social media is relatively new but becoming increasingly ubiquitous. The rapid and often variable growth of digital tools and interventions also increases the complexity of MM development.

The categories and labels for maturity stages vary, with five stages of maturity being most frequently used. The labels may lead to maturity assessments being interpreted as “report cards” rather than as reflection and quality improvement tools. This may detract from the need for mutually trusting and respectful partnerships to cocreate and realize collective health, technical, and social benefits from coordinated global management of pandemic and environmental challenges.^88,89

Conceptualization and development of DHMMs

The majority of papers (80%) used a conceptual framework, which they have developed or adopted/adapted from prevailing MMs, with many having their roots in the Capability Maturity Model Integration (CMMI).³ Other popular but domain-specific MMs include the EMR Adoption Model (EMRAM),⁹⁰ Control Objectives for Information and related Technology (CobiT) MM,⁴⁵ and Maturity Model for Enterprise Interoperability (MMEI, ISO-Norm 11354-2).³⁴ This diversity (Table 1) highlights the need for an inclusive DH metamodel that logically integrates the range of domain-specific MMs. It also highlights the need for a generic MM development methodology.

Development of content

The methodology to develop the content (items) of MMs usually includes the following phases within a lifecycle: Scope, Design, Populate, Test, Deploy, and Maintain.⁹¹ The scope and design is usually determined by domain experts, using Delphi group processes. The “populate and test” usually engages relevant practitioners and researchers in the domain. Data collected for testing may be qualitative such as a thick description,¹³ quantitative, eg, Likert scales, or usually, mixed qualitative and quantitative.^92,93Box 1 shows an example of MM content development.The small number of papers that addressed community or citizen engagement (n = 4) is indicative of the need for MMs in this domain, to encourage organizations to adopt a systematic approach to environmental and social responsibility.

Box 1.

The national nursing home information technology MM^55,93

The model includes measures for information technology uptake validated through a two-phase development process.⁷⁵ Phase 1 comprised a literature review resulting in a five-stage model. Phase 2 comprised a three-round Delphi process with nursing home experts. This produced a seven-stage model of IT maturity and a novel measurement tool.

The measures included 183 content items for 27 different content areas specifying the measure of information technology maturity. The majority of the content items (40%) were associated with information technology maturity stage 4, corresponding to facilities with external connectivity capability. Over 11% of the content items were at the highest maturity stage (stages 5 and 6). Content areas with content items at the highest stage of maturity are reflected in nursing homes that have technology available for residents or their representatives and used extensively in resident care.^55,93

Development of maturity stages

The number of maturity stages used in the literature range from 2 to 12, with most using a five-stage model. The number of stages and attributes of the stages of maturity are usually domain-specific and developed with expert content and methodological input. The basic principles of maturity staging include:

1. each stage is accumulative, incorporating attributes of the preceding stage and

2. there is a clear discrimination between maturity stages.⁸³

Specific attributes define each stage with an underlying logic that considers the context for the specific development status, circumstance or condition of an organization, a process, or a structure. The maturity attributes and stages are reviewed by experts iteratively as the development of the MM goes through the Populate-Test-Deploy cycle.⁹¹

The DHPMAT-MM used system performance and the extent of information sharing (interoperability) as generic attributes to assess the five stages of maturity.⁷ These attributes were applied at the micro (eg, personal and professional), meso (eg, organizational or community), or macro (eg, health system or national system) levels.

A life-cycle approach

An MM’s purpose evolves with the evolving maturity or age of the DH artifact, tool, procedure, or policy being assessed. A life-cycle approach to maturity modeling is important to address the differing purposes of an MM as it matures from being descriptive to prescriptive to comparative or integrative.^45,66,83,94 The lifecycle approach involves thick descriptions to gain a deep understanding of the DH domain, current maturity stage, and context to inform the prescription of guidance on feasible and sustainable strategies to improve the process and quality of digital services and any other identified gaps and weaknesses in the organization.

An example of the lifecycle approach is a longitudinal quality improvement study of 35 Swiss hospitals, using a comprehensive multiyear (2008–2014) data set obtained from a previously developed benchmarking and maturity assessment tool.^65,95 Exploratory-descriptive and path analyses to detect structural patterns and alternative explanations for the digital maturity led the authors to conclude that the most promising way to improve digital maturity was to invest in hardware and software because investments in personnel development or enhancements of operations and maintenance services did not show a significant relation. Digital maturity is a hospital's organizational asset that needs to be maintained and nurtured over time!⁶⁵

Another example of the lifecycle approach is the Enterprise Architecture Maturity Model (Box 2).

Box 2.

Lifecycle management of an enterprise architecture based on the TOGAF ADM*⁴⁵

The enterprise architecture (EA) domain exists to provide guidance on how to better align Business and IT. An MM for enterprise architecture in organizations is a governance instrument to analyze and evaluate the current state of affairs, as well as identify possible areas for improvement to facilitate the evolutionary reengineering of all functions related with the lifecycle of an enterprise architecture. This is done by enabling the benchmarking assessment and roadmap planning that is required as part of quality improvement.

Following a literature review of the existent models, Proença and Borbinha⁴⁵ developed an EA MM by collecting and evaluating the EA critical success factors. These include:

• Communication and common language;

• Business-driven approach;

• Commitment;

• Development methodology and tool support;

• EA models and artifacts;

• EA governance;

• Project and program management;

• Assessment and evaluation;

• IT investment and acquisition strategies;

• Skilled team, training, and education; and

• Organizational culture.

The authors concluded that there is no MM that fully considers all these critical factors. They propose to evaluate the EA MM iteratively through a multistep perspective, considering best practice and critical success factors in the EA domain, to confirm if the EA MM is useful and novel.

*The Open Group Architecture Framework (TOGAF) Architecture Development Method (ADM) is used to develop an IT architecture that meets the business needs of an organization.

Implementation and validation/evaluation of DHMMs

Implementation of MMs: from theoretical to pragmatic

Until recently, there has been little published about the implementation and effectiveness of applying MMs in complex environments like hospitals⁹⁶ and community-based facilities.^7,26 MMs have also offered frameworks to guide systematic development, implementation and improvement with regard to the assessed aspect.^82,83 Assessment of IT maturity and stage of maturity has important implications for understanding health service delivery systems, regulatory efforts, patient safety, patient engagement, and quality of care.

MMs and ontologies are separate but complementary strategies with the same purpose: to facilitate a holistic and real-world approach to the development, implementation and evaluation of DH solutions. The logical outcome of MMs and ontologies is an implementation framework and roadmap to achieve the desired objectives and maturity of the DH strategy. Box 3 describes the implementation of the use of an MM and framework.

Box 3.

The Sustainable intEgrated care modeLs for multimorbidity: delivery, Financing, and performancE (SELFIE) Horizon 2020 project

Background: SELFIE aims to produce evidence and applicable policy advice on ICC programs for people with multimorbidity. SELFIE is the maturity assessment framework used to analyze the Digital Health Transformation of Integrated Care in Europe.¹³ A wide range of digital technologies was assessed for their successful adoption and implementation and effectiveness in supporting healthcare transformation. This longitudinal implementation and assessment study identified common facilitators of and barriers to the reach, adoption and maintenance of DH implementations to achieve and sustain improved health outcomes, cost-efficiencies, patient satisfaction and provider well-being in 17 selected integrated chronic care (ICC) programs from 8 European countries (Austria, Croatia, Germany, Hungary, the Netherlands, Norway, Spain, and the United Kingdom).

Methods: Thick descriptions based on program document analyses and semistructured interviews with 233 relevant stakeholders—professionals, providers, patients, carers, and policymakers. The overarching analysis focused on the use of DH tools and program assessment strategies.

Findings: DH tools are implemented in all countries, but different levels of maturity were observed among the ICC programs. Only a few programs have well-established strategies for a comprehensive longitudinal assessment. There is a strong relationship between DHM and proper evaluation strategies of the ICC program. Despite the heterogeneity of DHM across the countries, most ICC programs aim to evolve toward a digital transformation of integrated care, including implementation of comprehensive assessment strategies.

Conclusions: The evolution of DH tools alongside clear policies toward their adoption will facilitate regional uptake and scale-up of services with embedded DH tools.

MMs to guide the development of implementation frameworks

Numerous MMs have been developed to support IT management in a broad range of application areas, including: holistic assessments of IT management, appraisals of specific areas such as business process management and business intelligence, IT service capability, strategic alignment, innovation management, program management, enterprise architecture, and knowledge management maturity. However, most of these models were simply a means to position the selected unit of analysis on a predefined maturity scale.⁹⁷

Recent regulations, mandates, policies, and guidelines set forth by the US government, federal and other funding agencies, scientific societies and scholarly publishers, have imposed stewardship requirements on digital scientific data. Maturity assessment models have been developed to support the need for a formal approach to stewardship activities that supports compliance verification and reporting. Meeting or verifying compliance with stewardship requirements requires assessing the current state, identifying gaps, and, as a logical sequence, defining a roadmap for improvement.⁹⁸

MMs to guide assessment and evaluation

Maturity assessment of the competency, capability, and level of sophistication of a selected domain may use MM and non-MM-based methods. The MM measurement approach is less diverse but more targeted to specific objectives as it is constrained within the model. It usually defines a basic set of maturity levels (listed as rows) defined by a set of attributes listed as columns in the model matrix. The underlying program logic is usually framed within an organization or enterprise.

Standardized DHM staging measures and indicators are needed to help draw congruent and similar comparisons among settings where the maturity is being measured routinely or episodically. Data collected for the maturity assessment may be qualitative, quantitative or mixed and may be collected prospectively into the future (a priori) or retrospectively (a posteriori) to evaluate current or future systems. Box 4 describes an MM-based assessment method.

Box 4.

An MM to evaluate the knowledge management capabilities (KMC) of an organization

The evaluation of KMC^21,99 to capture, transfer, and disseminate data, information, and medical knowledge of a healthcare facility is essential because they are increasingly a nexus for exchanging detailed knowledge and information at the micro (interprofessional/interpersonal), meso (inter-organizational) and macro (enterprise-wide) levels within a health system. This is a key component of essential foundation “readiness to share information.”

There is an increasing demand to assess the efficiency and effectiveness of and optimize support for delivery of routine health services and medical procedures by an increasing number of professional disciplines. The unique knowledge emerging from the interactive delivery of services during the patient-clinician relationship needs to be captured, secured, shared, and used to improve the knowledge base and their operational effectiveness for all professional disciplines from all health related sectors. Finally, patients are increasingly more demanding regarding the quality of the care they receive along with medical information on their illness.

The KMC MM developed qualitatively includes three dimensions:

1. knowledge infrastructures in knowledge management;

2. knowledge management process; and

3. knowledge management competency.

The maturity of these three dimensions of knowledge management is assessed as part of the evaluation of the implementation of digital knowledge management tools and KMC of the actors. The maturity assessment framework has three constituent components:

1. a framework characterizing the lens through which individuals view situations;

2. a framework for characterizing how individual habits change; and

3. a framework for characterizing the manner in which emergent understandings are consolidated into existing knowledge and knowledge structures.

The KMCMM provides for the identification of factors that influence the nature and effectiveness of the use of KMC in healthcare facilities as well as an MM for evaluating health organizations in how they meet the maturity benchmarks for information flow and management of their KM.

The “spread and scale-up” of an MM approach to DH

Examples of “spread and scale-up” were rare because mainstreaming has been constrained by the diversity of MMs. This conceptual diversity among many similar or overlapping specialized domain-specific MMs has led to a degree of conceptual confusion among the implementers and evaluators of DH in all countries.¹⁰⁰ This confusion adds to the cognitive overload on human actors in DH by the plethora of usually incomprehensible toolkits, manuals, and other documentation for artifacts and tools.^7,101,102 This leads to poor implementation and lack of adoption of DH by local citizens and health workers. This poor “spread and scale-up” is common even among the more established management and HIS research programs from more developed countries.^11,18,34,62

Nevertheless, an example of a temporal “spread and scale-up” of an MM approach to DH is the digital maturity of 35 Swiss hospitals since 2008. A comprehensive multiyear (2008–2014) data set was created from the use of a benchmarking and maturity assessment tool.^65,95 This longitudinal data-driven quality improvement and monitoring program enabled exploratory-descriptive and path analyses methods to detect structural patterns and possible explanations for the digital maturity. These Swiss hospitals were found to have a strong internal focus and were reactive in their approach to digitalization. Digital maturity was found to be a relative and subjective construct that either improved or worsened over time, was bound to perceptions of health professionals, and seldom reached a final stage. This dynamic nature of the indicators of digital maturity increases the complexity of the evaluation methodology. The variance in health information technology (HIT) appraisal of hospitals was due to several factors, such as enhancements in the eco-system, changes in the user base, or unforeseen and unprepared system adaptations. From the quality improvement perspective, the authors concluded that the most promising way to improve digital maturity was to invest in hardware and software because investments in personnel development or enhancements of operations and maintenance services did not show a significant relation. Digital maturity is a hospital's organizational asset; this is dynamic and needs to be maintained and nurtured over time!

A metamodel is needed to harmonize and integrate specialized MMs

The diversity of MMs is a response to the multiple needs of organizations in the face of competition and resource constraints. While a choice of MMs for organizations is good, the resultant quagmire of standards and models¹⁰³ is also a challenge to good management, governance, research, and quality improvement of the DH system. The variability in methods and domains creates conceptual and cognitive confusion among developers, implementers, and evaluators of DH policies, strategies, programs, and tools,¹⁰⁰ especially in poorly resourced low- and middle-income countries.^7,101

This diversity and specialization raises the need for a metamodel to logically “harmonize and integrate” the diversity of MMs that have been developed in the DH enterprise. This metamodel, with guidance and tools to support the integration of multiple MMs, is a potential solution.¹⁰⁴ The CMMI, whose models are collections of best practices that help organizations to improve their processes, is a federation of some existing MMs.¹⁰⁵ While it is not a metamodel, it is a step in the right direction.

Harmonization of multiple MMs for a broader purpose within the organization or across the enterprise require capability maturity in the assessment of benchmarks for data quality,^33,106 standards for interoperability,^34,107,108 and the fitness for purpose of the enterprise platform.^109,110 As knowledge exchange and data/information sharing is not possible without an enterprise architecture and interoperability standards, the need for MMs in these areas is apparent.^33,106

The DHPMAT-MM approach can advance this integration as a metamodel to harmonize the specialized MMs by guiding:

1. the development of a functional classification of the specialized artifacts being assessed by the diverse MMs within the domain, organization, or enterprise;

2. the selection of the specialized MMs to harmonize and integrate into the DHPMAT-MM; and

3. the systematic implementation the DHPMAT-MM into management and governance within the organization and across the enterprise.⁷

Some guiding principles

The synthesis of the themes identified some high level properties common to the development, implementation and evaluation of capability MMs. These reflections may be viewed as guiding principles, viz:

1. A dynamic lifecycle approach to digital capability maturity, which is:

   1. responsive to environmental changes and may improve or worsen over time;

   2. accumulative, incorporating the attributes of the preceding stage; and

   3. sequential, where no maturity stage must be skipped.

2. Sociotechnical quality improvement of the digital artifact and MM, which includes;

   1. investing in the organization’s human, hardware and software resources and

   2. a need to engage and improve the DH competencies of citizens.

1. A dynamic lifecycle approach to digital capability maturity

The principles of maturity stage development stipulate that each discrete stage is accumulative, incorporating attributes of the preceding stage.⁸³ The assumption is that maturation is linear and sequential.¹¹ An example is the Hospital Information System Maturity Mode (HISMM), which defines six maturity stages and recommends against skipping any intermediate stages. The rationale is that optimum results are unlikely if practices in the lower stages remain unfulfilled.

This linearity in MMs is analogous to the Maslow’s “hierarchy of needs,”^111,112 which theorizes that the appearance of one need usually rests on the prior satisfaction of another biologically, culturally, or situationally determined need. Failure to achieve prior levels may result in unanticipated adverse consequences in the next. A clinical example is a reported increased risk of falls associated with the introduction of person-centered care (PCC) in aged care facilities. Perhaps PCC, a higher level need, should be implemented only after safety and physiological needs are met.¹¹³

Nevertheless, in practice, maturity stages are often” worked around” for practical and business process management reasons.¹⁴ When filtered through the DHPMAT-MM,⁷ many past and existing DH projects were found to be maturity stage 4 (optimizing) quality improvement projects involving mobile health apps that have not maximized potential benefits, especially where they involve integrated and coordinated care.^7,114 This is because the noncompletion of maturity stage 3 (standardized) meant a lack of a stable standards-based enterprise architecture and platform to adequately support interoperable tools and data that are essential to support integrated care cost-effectively. In addition to compromising cost-effectiveness and quality, practitioners using “work arounds” need to be aware of unanticipated safety issues through working with compromised data quality and interoperability.

2. Sociotechnical quality improvement over the lifecycle of the digital artifact and MM

Inherent in the capability maturity construct is quality improvement over the lifecycle of the data resource being studied. This mandates that DHM assessment draws on multiple disciplines, including engineering, psychology, sociology, and organizational behavior.¹¹⁵ These developments are essentially evolutionary theory as applied to the digital artifacts, and influenced by technology and management theory. The two key variables are the identification of the data resource elements and the conceptualization of their growth through inputs and interventions over time. Maturity is achieved when the data resource reflects the organization and its information flows. In this reflection and quality improvement, it is important to remember that digital maturity may be constantly in flux.

We will also label the five maturity stages of the DHPMAT-MM as “Assessing—Getting control—Standardising—Optimising—Innovating” to emphasize the quality improvement journey across the sociotechnical DH foundations.

CONCLUSIONS

The diversity in MMs and the variability in their methods and domains can create conceptual and cognitive overload among the stakeholders of DH policies, strategies, programs, and tools.⁷ The DHPMAT-MM has the potential to unify domain-specific MMs, and to align these MMs to the purposes and objectives of the enterprise architecture as it evolves over time. This dynamic metamodel—a DHMM—should also guide the coimplementation and coevaluation of MMs for fidelity, validity, relevance, and adoption in a culturally respectful manner.

AUTHOR CONTRIBUTIONS

• STL conceptualized and designed the work, collected and interpreted the data, drafted the work critically for important intellectual content, approved the final version, and is accountable for all aspects of the work.

• MAG made substantial contributions to the conception and design of the work, collected and interpreted the data, revised the work critically for important intellectual content, approved the final version, and is accountable for all aspects of the work.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

DATA AVAILABILITY

All data are contained within the manuscript and associated materials.