Abstract
Introduction
Countries need to determine their level of digital health capability maturity to assess and mobilize their knowledge, skills, and resources to systematically develop, implement, evaluate, scale up and maintain large-scale implementations of standards-based interoperable digital health tools.
Objective
Develop a Digital Health Profile and Maturity Assessment Toolkit (DHPMAT) to assist Pacific Island Countries (PICs) to harness digital tools to support national health priorities.
Materials and Methods
A literature review guided the development of the conceptual framework to underpin the DHPMAT. Key informants collaborated to collect key digital health features and indicators to inform their country’s digital health maturity assessment. The DHPMAT was tested with country stakeholders at a Pacific Health Information Network workshop in 2019.
Results
A comprehensive list of indicators to describe country digital health profiles (DHP). A digital health maturity assessment tool that uses criteria codeveloped with country stakeholders to assess essential digital health foundations and quality improvement. DHPs created and maturity assessed and packaged into individualized DHPMATs for 13 PICs. PIC users perceived the DHPMAT as useful, especially the congruence with the 2017 WHO WPRO Regional Strategy but noted a “cognitive overload” from a plethora of complex digital health toolkits.
Conclusions
The cocreation approach optimized currency, accuracy, and appropriateness of information in the DHP, understanding, and use of the DHPMAT to facilitate informed iterative discussion by PICs on their digital health maturity to harness digital tools to strengthen country health systems. The DHPMAT can rationalize the choice and use of existing tools and reduce cognitive overload.
Keywords: digital health, informatics, cocreation, implementation, evaluation, maturity assessment
INTRODUCTION
The 2018 World Health Assembly (WHA) urged Member States to:
“… assess their use of digital technologies for health, including health information systems at the national and subnational levels, in order to identify areas of improvement, and to prioritize the development, evaluation, implementation, scale-up and greater utilization of digital technologies, as a means of promoting equitable, affordable and universal access to health for all, including the special needs of groups that are vulnerable in the context of digital health.”1
In 2018, the WHO Regional Committee for the Western Pacific endorsed the Regional Action Agenda on Harnessing eHealth for Improved Health Service Delivery (eHealth Regional Action Agenda).2 This blueprint for action included the development of country digital health profiles to provide summarized insights to governments, policy makers, administrators, and stakeholders on the maturity of essential digital health foundations to address national health priorities.3
Understanding their digital health maturity will enable countries to meaningfully align their knowledge, skills, and resources to systematically develop, implement, and evaluate standards-based interoperable digital health systems and programs to support and sustain their health priorities. A recent review of 14 maturity models in information systems and technologies in health care highlighted the need for a more holistic approach to integrate the current approaches to maturity models, which are either specialized or generalized.4 This holistic approach should be contextualized within the national health priorities.
We therefore aimed to develop a Digital Health Profile and Maturity Assessment Toolkit (DHPMAT) to assist Pacific Island Countries (PICs) to assess their digital health capability maturity to implement and evaluate a national digital health program.
MATERIALS AND METHODS
This included a literature review of maturity assessment tools (MATs) to develop the common understanding and conceptual framework to guide the DHPMAT.5 MATs were also collected through the WHO Institutional Repository for Information Sharing and included if they had been evaluated or field-tested.
Definitions to establish a common understanding
Digital health
Digital health is “a broad umbrella term encompassing eHealth* as well as developing areas of advanced computing sciences in the fields of ‘big data,’ genomics, and artificial intelligence” (WHO EB142/20, 2017).6 Digital health is consumer-centric through the use of smart phones and wearable sensors connected through the Internet of Things.7
*eHealth was defined as the “cost-effective and secure use of ICT [information and communications technology] to support health and health-related fields, including health services, health surveillance and health-related literature, education, knowledge and research” (WHA58.28 on e-health, 2005).
Digital health profile
The digital health profile (DHP) includes summarized information compiled from available data about the country context, health priorities, digital health developments, and quality improvement programs. Digital health indicators take on different meanings and significance from the perspectives of the patient, health professional, organization, or system. DHPs may be sufficiently granular for in-depth assessments of digital health maturity at subnational levels.
Digital health maturity models
A maturity model is a set of structured levels that describe organizational behaviors, practices, and processes that reliably and sustainably produce required outcomes.4,8 It measures the ability of an organization to continuously improve in specific dimensions until it reaches the desired level of maturity.4 Factors that influence the achievement of mature, interoperable information systems are identified and quantified at each maturity level to guide and monitor the improvement in maturity. Like a roadmap, the maturity model describes the processes that lead to better outputs and outcomes from 1 level to the next. Inherent in maturity assessment is the need for measurement, evaluation, and quality improvement tools and protocols.
Developing the maturity assessment conceptual framework for the DHPMAT
The conceptual framework included the 3 dimensions—digital health foundations, maturity assessment and quality improvement—described below.
Essential digital health foundations to support health services
The health services oriented digital health framework (Figure 1) is central to 2017 WPRO eHealth Regional Action Agenda to harness digital tools that strengthen health systems and improve access, equity, safety, and quality of health services.2 Our literature review on MATs supported a comprehensive approach in designing the DHPMAT to assist PICs to harness digital health to improve health services from the perspectives of citizens, communities, health professionals (clinical, managerial and technical), health organizations (technical, operations and governance), and the national health system.
Figure 1.
Health services oriented digital health framework.
The essential digital health foundations include: essential ICT infrastructure, essential digital health tools, readiness for information sharing and health system adoption. The elements of the foundations were derived from the Atlas of eHealth Country Profiles 2015,9 WHO-ITU National eHealth Strategy Toolkit,10 and WHO Guideline on Digital Health Interventions.11 They include leadership and governance; strategy and investment; legislation, policy, and compliance; workforce standards and interoperability; infrastructure; services, applications, and tools such as telehealth, electronic health records (EHRs), eLearning, mHealth, social media, and big data.
Digital health maturity assessment tools
To assist country level assessment of its capability maturity to implement and evaluate a national digital health program, the maturity assessment tool (MAT) must be sufficiently comprehensive and flexible to capture the perspectives of health systems, organizations, professionals, and consumers. Four MATs identified in the literature review were selected to illustrate the requirements for the health system, organization, and individual users (Table 1), including the following:
Table 1.
Digital health maturity assessment models used in developing DHPMAT
| Maturity assessment model | Focus of model | Maturity Categories | Maturity descriptors |
|---|---|---|---|
| Informatics Capability Maturity Model (ICMM) | Health organization |
|
General descriptors with examples for 5 levels of maturity: basic, controlled, standardized, optimized, and innovative. Respondents reflect on their digital health profile and choose a maturity level. |
| Global Digital Health Index (GDHI) | National digital health system |
|
Prescriptive descriptors with respondent ticking 1 of 5 specific statements for each category & subcategory ranked according to levels of maturity. |
| Health Information Systems Interoperability Maturity Toolkit (HISIMT) | Health organization (technical & operational) |
There are a number of subcategories |
Prescriptive descriptors with respondent ticking “yes/no” for 1 of 5 specific statements for each category & subcategory to ascertain 5 levels of maturity: nascent, emerging, established, institutionalized, or optimized. |
| Health Information System Stages of Continuous Improvement Toolkit (HISSCIT) | Health organization (technical & operational) |
|
Prescriptive with respondent ticking “yes/no” for 1 of 5 specific statements for each category & subcategory to ascertain 5 levels of maturity: Emerging/Ad hoc, Repeatable, Defined, Managed, or Optimized. |
Global Digital Health Index (GDHI) has a national focus and a framework with 7 categories: Leadership and governance; Strategy and investment; Legislation, policy and compliance; Workforce; Standards and interoperability; Infrastructure; and Services & applications.12
Informatics Capability Maturity Model (ICMM) has an organizational focus and a framework with 5 dimensions: Managing information; Using business intelligence; Using information technology tools; Aligning business and informatics; and Managing change.13,14
Health Information Systems Interoperability Maturity Toolkit (HISIMT) has a focus on interoperability and a framework with 3 domains: Leadership and governance; Human resources; and Technology.15
Health Information System Stages of Continuous Improvement Toolkit (HISSCIT) has a focus on continuous improvement and a framework with 7 categories: leadership & governance, management, ICT infrastructure, systems, data interoperability, data quality, and data use.16,17
There is significant overlap in the dimensions/categories/domains used in these MATs. They may be described differently depending on the purpose or whether it applies to a micro (health facility), meso (health district) or macro (health system) organization.
Maturity is assessed for various elements within each dimension/category/domain. The descriptors for each maturity level may be prescriptive as in ticking 1 of 5 specific statements on maturity of the categories in the GDHI (Box 1); or a “yes/no” answer to a number of specific statements to ascertain if the maturity was nascent, emerging, established, institutionalized, or optimized in the HISIMT (Box 2) or Emerging/Ad hoc, Repeatable, Defined, Managed or Optimized in the HISSCIT. The ICMM provides general descriptors and examples for 5 levels of maturity through basic, controlled, standardized, optimized, to innovative (Box 3).
The GDHI is too high-level to be useful operationally. On their own, the more specific MATs described do not provide all the information required to assess the overall DH maturity at the required level of granularity. Appropriate concepts from existing MATs, particularly the 4 analyzed, are incorporated into the DHPMAT framework. Where gaps are identified, the DHPMAT can address them or suggest specific validated MATs to conduct the specific operational or technical assessment. The DHPMAT can be viewed as an overarching framework to guide the evidence-based use of specific MATs from the plethora found in the literature review.
Context, implementation, monitoring, and quality improvement
Digital health interventions complement and enhance health system functions through accelerated access to and exchange of information. They do not replace fundamental health systems components such as the health workforce, financing, or governance. The WHO guideline stressed that digital health investments must be evidence-based and include an assessment of risks against comparative options.11
Digital health implementation is affected by background cascading challenges, comprehensively summarized by the non-adoption, abandonment, scale-up, spread, and sustainability (NASSS) framework.18 The NASSS domains includes the condition/illness, technology, value proposition, adopters, organizations, institutional and societal context, and interactive adaptation between all these domains over time. The NASSS asserts that: simple (straightforward, predictable, few components) interventions are easy to implement while complicated (multiple interacting components) or complex (dynamic, unpredictable, not easily disaggregated into constituent components) ones are difficult to implement or sustain.
The Normalization Process Theory (NPT) guides assessment of readiness to adopt and use digital health interventions.19 The NPT works well with the NASSS framework to enable managers, clinicians, and other users to understand the conceptual and contextual factors in successful implementation and evaluation of new technologies and interventions in health care.12 They can inform the design of a new technology, identify technological solutions that are difficult to scale up and sustain, plan the implementation and rollout, and explain and learn from program failures or successes.
Finally, measurement and monitoring are essential because quality improvement is implicit in all maturity models. Maturity assessments of systems and organizations become more accurate as they move up the maturity continuum. A national network of digital health profiles and maturity can highlight geographic and demographic variations over time, enabling targeted planning and equitable resourcing of digital health services. With careful planning, a comparative effectiveness research program can be embedded in the national health strategy to relate digital health maturity to access, equity, safety, and quality of care. The WHO Practical Guide20 and mHealth Assessment and Planning for Scale toolkit21 guided the quality improvement, monitoring, and evaluation components of the DHPMAT.
Developing the DHPMAT
The maturity assessment conceptual framework, especially the WPRO eHealth Regional Action Agenda2 and the ICMM14 guided the development of the DHPMAT. The WHO Collaborating Centre on eHealth, WHO WPRO and key stakeholders including WHO Department of Pacific Support, Pacific Health Information Network (PHIN), Pacific Community, and key informants from WHO country offices adopted a systematic cocreation22 approach to:
Cocreate the Digital Health Profile (DHP)
Quantitative and qualitative country indicators, including those from the International Telecommunication Union publications,23,24 were collected and checked with key informants to confirm the accuracy, relevance, importance, and uniqueness to the PIC.
Cocreate the Digital Health Maturity Assessment Tool (DHMAT):
The DHP indicators were interpreted based on the maturity of the 4 essential digital health foundations as assessed by the DHMAT.
Key informants refined and verified the indicators and DHP iteratively. This included data interpretation (strengths and challenges) and drafting the key maturity levels for individual PICs.
Codevelop local digital health maturity assessment criteria and the DHPMAT
Multiple versions of DHPMAT were developed and discussed iteratively.
An initial version of the DHPMAT was tested with key informants before final testing at a PHIN workshop. A feedback questionnaire with focused and open questions and interviews collected both quantitative and qualitative data.
The collective creativity was harnessed with digital technologies, using discursive processes of sharing multidisciplinary and multifaceted data, information, and knowledge. Participants in this cocreation community played multiple roles in socializing and evolving the nature and quantity of the content based on their perceptions and contexts.22 The emerging social network defined and developed the DHPMAT as a knowledge resource and social capital that can be accessed and used for its intended purpose.
RESULTS
Output: Indicators for digital health profile and maturity assessment
The following indicators were determined for the essential digital health foundations (Table 2):
Table 2.
Digital Health Profile (DHP) Indicators used to assess Digital Health Maturity (DHM)
| Element of DHPMAT | DHP Indicators used to assess digital health maturity |
|---|---|
| Country context | 17 indicators to describe the country’s demographics, politics, socioeconomics, and general rule of law |
| ICT infrastructure | 21 indicators to assess the overall ICT development (ICT development index rank and WEF Network readiness index)) with focus on ICT coverage, ICT access, ICT affordability, and ICT resilience. How distributed is the national digital health program? |
| Essential digital health tools | 39 indicators to assess the current establishment and use of essential digital health tools including unique identifier, health information system (EMR/EHR), clinical decision support, telemedicine & mHealth, social media, eLearning, and big data analytics. How has the national health data asset (eg, registers and EMRs/EHRs) been developed and maintained? |
| Readiness for information sharing | 27 indicators to assess the readiness in terms of an interoperability framework, architecture, standards, data quality, and legal frameworks. How are health and other information collected and managed to a standard? |
| Health system adoption | 31 indicators to assess leadership & governance; funding, current strategy, and investment in digital health; digital and health literacy; capacity building for digital health development and deployment; services and applications available. How and where have digital health literacy projects been deployed? Assessment of the complexity and complicatedness of digital health intervention and project. |
| Quality Improvement, Measurement, Monitoring, and Evaluation (QIMME) |
Describe QIMME programs which can range from single-site, ad hoc, and descriptive evaluation to multisite implementations with logical comparative effectiveness research across various sites to evaluate large-scale implementations with assessment of financial and economic value and measurements of impacts (processes and outcomes). Indicators to assess and monitor the improved sharing of interoperable and fit-for-purpose data (data quality), quality of care, equity, and access to care. Indicators to assess and monitor the RE-AIM elements of implementation: Reach, Effectiveness, Adoption, Implementation, Maintenance (individual, community, and organization). |
ICT infrastructure: electric power, mobile signal and internet coverage, ICT conditions and user access to hardware in different settings, prevailing use of digital technologies.25
Essential digital health tools to collect, record, store, and use personal health information accurately and securely, including the use of patient registries, EHRs/EMRs, electronic decision support and mobile and telehealth. A unique identifier for individual and health service is essential to underpin the smooth functioning of digital health.
Readiness for information sharing: Enterprise architecture, standards, interoperability, privacy and security are important to enable information exchange and preventing harm from errors and misuse of information. Privacy regulations, guidelines and training on data curation and quality management, and continued investment in cyber defense are important enablers.
Health system adoption: Policies and strategies to support broader health system adoption including national digital health policies and strategies, suitable governance arrangements, workforce capacity, and partnerships and engagement with key stakeholders.
The concepts and indicators from existing MATs, particularly the tools shown in Table 1, are incorporated into the DHPMAT framework under the relevant essential digital health foundation. Which indicator gets emphasized in the DHPMAT is a result of the cocreation and dialogue with key informants and country WHO staff. More than that, the cocreation enabled a more relevant and accurate country digital health profile and indicators.
Output: Quality improvement, measurement, monitoring, and evaluation (QIMME)
The publication Monitoring and Evaluating Digital Health Interventions20 emphasizes a common quality improvement framework for evaluation to generate evidence required for decisions on investments in digital health. Selected digital tools should have good evidence of effectiveness derived from local, national, or international studies. QIMME is considered mature when it is embedded in programs, and there is a coordinated measurement, monitoring, and quality improvement program, along with evaluation of the implementation for process efficiencies and impacts.
Output: the digital health profile & maturity assessment toolkit (DHPMAT)
The DHPMAT includes:
The Digital Health Profile (DHP) to be used as the information source for maturity assessment. An example of a DHP is shown as Supplementary File 1.
The Digital Health Maturity Assessment Tool (DHMAT) to guide an objective self-assessment of the strengths and weaknesses of the essential digital health foundations—ICT infrastructure, essential digital tools, digital health adoption and information sharing (Figure 2a and 2b). The DHMAT uses generic descriptors of the maturity levels:
Figure 2.
(a) Digital Health Maturity Assessment Tool (DHMAT) with descriptors and examples for each maturity level. (b) A Quality Improvement, Measurement, Monitoring, and Evaluation (QIMME) maturity scale.
Basic → controlled → standardized → optimized → innovative
Descriptors and examples based on information distilled from various DHPs are included as guidance for allocating the level of maturity for each essential foundation.
The Quality Improvement, Monitoring, Measurement, and Evaluation (QIMME) scale (Figure 2b).
A template to align relevant key information about the country context—health priorities, strengths, and challenges—with digital health maturity to facilitate informed discussion on digital health tools to adopt, adapt, or develop in context (Figure 3). The objective is realistic implementation plans with embedded quality improvement and evaluation. Important principles are:
evidence-based and implementable interventions,
simple, measurable, achievable, realistic, and time-bound (SMART) objectives, and
an implementation and evaluation plan with a relevant program logic model.
Figure 3.
An activity planning guide to achieve the desired Digital Health Maturity to support the relevant health priority.
Figure 4 is a visual of the journey of a DHPMAT user from cocreating the DHP to using the DHPMAT to guide the development of an implementable and sustainable digital health program.
Figure 4.
An example of a user journey with the DHPMAT.
Results of user testing of the DHPMAT
Digital health profiles and maturity assessments were cocreated for 13 PICs—Cook Islands, Fiji, Federated States of Micronesia, Kiribati, Marshall Islands, Nauru, Palau, PNG, Samoa, Solomon Islands, Tonga, Tuvalu, and Vanuatu. This iterative process produced a version of the DHPMAT for testing with country stakeholders at a PHIN workshop in Noumea in July 2019. Participants (n = 11) were provided with the DHPMAT and user guide a week before the workshop, which they used and, subsequently, provided feedback.
The participants reported the DHPMAT as useful, relevant for their country, had easy-to-understand-and-follow instructions, and had a logical rationale underpinning its use (Table 3). The link to the WPRO eHealth Regional Strategy, particularly the “thought-provoking” questions and discussion points, was scored highly. The content of the DHPs were considered relevant and important but could be improved in accuracy and completeness. Participants reported a need to better understand the rationale for the indicators and levels of maturity, suggesting training and advocacy for the use of DHPMAT. The higher than average scores for the design, structure, and usefulness of the DHPMAT is encouraging, reinforcing the adoption of “cocreation” to ensure accuracy and relevance of the digital health profile and emphasizing the need for a “hands-on” sociotechnical and capacity-building approach to maturity assessment to guide digital health strategy development.
Table 3.
Perceptions of the DHPMAT (5-point Likert scale)
| Element of the DHPMAT assessed | Average score |
|---|---|
|
1. Rationale & Methods |
3 |
|
i. The content is easy to understand |
4 |
|
ii. I know the reasons why the DHPMAT was developed |
3 |
|
iii. I know how the DHPMAT was developed |
2 |
|
iv. The rationale also addressed my country’s priorities |
3 |
|
v. The link to the WPRO Regional Strategy is important |
4 |
|
2. User Guide |
4 |
|
i. The instructions are easy to follow |
4 |
|
ii. The recommended way to use the DHPMAT is logical |
4 |
|
3. Country Digital Health Profile |
3 |
|
i. The content is accurate |
3 |
|
ii. The content is relevant and important |
4 |
|
iii. The content is complete |
3 |
|
4. Digital Health Maturity Assessment Tool |
3 |
|
1. The maturity assessment tool is easy to use |
3 |
|
ii. The 4 essential digital health foundations are logical |
3 |
|
iii. The attributes allocated to each foundation are correct |
3 |
|
iv. The 5 digital health maturity levels are logical |
3 |
|
v. The descriptors for each maturity level are logical |
3 |
|
vi. The ‘thought-provoking’ questions are useful |
4 |
|
vii. The ‘discussion’ points are useful |
4 |
|
viii. The ‘examples’ are appropriate |
3 |
|
5. General Perceptions |
3 |
|
i. The overall framework covers all aspects of digital health |
3 |
|
ii. The DHPMAT was easy to use |
3 |
|
iii. The DHPMAT is useful and will meet its objectives |
4 |
|
iv. I am already familiar with the recommended resources |
3 |
|
v. This is relevant and useful for country |
4 |
Qualitative feedback was generally constructive and pragmatic. Examples include the following:
It's good for a baseline information to compare with the reality on the ground—Respondent 2
An insightful comment concerned the “cognitive overload” related to the many digital health toolkits distributed by agencies in the PICs.
Considering the number of toolkits used in the country for the same space, we will park it for now until further consultation—Respondent 5
Based on this user feedback, the DHMAT User Guide and rationale was revised to be more explicit and understandable, including how the DHPMAT may rationalize the choice and use of existing digital health toolkits.
DISCUSSION
A collaborative and systematic maturity-based approach to making decisions to invest, develop, implement, evaluate, and maintain digital health programs for identified priorities and tailored to local contexts and opportunities is a logical one. This is particularly true of small PICs, where limited service capacity, unreliable basic and ICT infrastructure, and poor access/affordability limit the range of digital tools that can be used successfully. These limitations along with the reported “cognitive overload” from the plethora of largely unevaluated digital health toolkits distributed as part of various aid programs may explain why the participants were tentative in their perceptions. As a tool to focus and base a country’s digital health strategy on its maturity level, the DHPMAT can facilitate the informed choice and use of specific tools for the assessment of specific digital health foundations. It is not a “one-size-fits-all” but a logic-based approach to the choice of tools. The tentativeness also indicated a need for a well-tested DHPMAT that is easy to understand and supported by well-trained facilitators and change agents.
Cocreation recognizes the fact that the content of the DHPMAT will evolve with time and improved digital health maturity. Countries need to own and proactively keep the DHP up-to-date to enable regular and accurate maturity assessment. Engagement with the DHPMAT cocreation network is important to ensure that maturity indicators selected are the most appropriate for planning, developing, implementing, and evaluating digital health programs—not just for reporting. The DHPMAT should therefore be informed by sufficiently granular data to support program implementation and evaluation, even down to the facility level. Cocreation also improved perception of the DHPMAT as a tool to facilitate ongoing discussions on digital health programs.
Participants in the cocreation increased their awareness of the link between the DHPMAT and WPRO eHealth Regional Strategy.2 There was an explicit congruence between the 5 DHPMAT maturity levels and the 3 stages of development (initial, developing, and advanced) described in the WPRO strategy. The strategies proposed by WPRO for each stage also made sense at the relevant maturity level. Countries with basic or controlled maturity levels are at the initial stages of development. They could identify “leapfrog” opportunities to leverage the wireless telephone infrastructure into SMS-based telehealth services. They could also digitize and standardize existing databases (eg, disease registers) or implement a Universal Health Identifier system. Countries with controlled or standardized maturity levels could focus on improving coverage and quality of health services through mHealth/telehealth; update enterprise architecture and interoperability standards; conduct QIMME; and systematically phase in more complex digital health interventions such as EHRs by module or site. Countries with optimized or innovative maturity levels could foster innovative eHealth apps to improve services; improve interoperability and data quality; strengthen privacy legislation, regulation and enforcement; and mentor countries with lower digital health maturity and share technical expertise and resources across sectors to achieve economies of scale.
Grounding the cocreation of the DHPMAT in the context of the eHealth Regional Strategy ensures relevance and facilitates the scale-up, testing, and further refinement of the DHPMAT to facilitate the in-country dialogue on harnessing digital tools to strengthen health systems and achieve universal health coverage and sustainable development goals.
Limitations
There are deficiencies in the availability and quality of data on the essential digital health foundations, current digital health and quality improvement programs, and funding for digital health in the PICs. However, the tools and protocols for data collection, validation, and quality assurance are improving, especially in PICs actively participating in the cocreation process.
CONCLUSION
Quality improvement is implicit in maturity assessment, emphasizing the need to embed measurement, monitoring, and evaluation into the digital health program. The cocreation approach to developing and testing the DHPMAT ensured currency and appropriateness of information collected and understanding and use of the DHPMAT to facilitate informed discussion by PICs on their digital health maturity. Reducing the cognitive overload from the multitude of disparate tools made available to various development agencies is an important consideration. User testing and post-workshop discussions within the social network developed from the cocreation approach indicated the need for sociotechnical change management strategies, including advocacy and trained DHPMAT facilitators to systematically implement and evaluate digital health strategies in the PICs. The DHPMAT cocreation approach and the certainty of changing technology and country contexts means that the DHPMAT will certainly be evolving through an ongoing adaptation process as it begins to be used in other countries and settings.
FUNDING
This work was supported by the WHO Western Pacific Regional Office (WHO Reference # 2019/892799-0; UNSW Grant #RG191104).
AUTHOR CONTRIBUTIONS
STL and JG conceptualized and guided the project. All authors contributed substantially to the project implementation: data acquisition, analysis, and interpretation; and iterative enhancement of conceptual framework. STL wrote the first draft. All authors reviewed and revised it carefully for intellectual content, approved the final draft, and agreed to be accountable for the accuracy and integrity of the work.
SUPPLEMENTARY MATERIAL
Supplementary material is available at Journal of the American Medical Informatics Association online.
Supplementary Material
ACKNOWLEDGMENT
The authors thank the cocreators in the participating PICs and Ms Katri Konti (WHO Pacific Technical Support) for collaborative input into and interpretation of the digital health profile; and Dr Myron Godinho (UNSW Sydney) for general editorial review of the paper.
CONFLICT OF INTEREST STATEMENT
None declared.
Box 1: The Global Digital Health Index categories and maturity descriptors
With the GDHI, there are 5 response options to a “Standards and Interoperability” question: Is there a national digital health (eHealth) architectural framework and/or health information exchange (HIE) established?
There is no national digital health (eHealth) architectural framework and/or health information exchange (HIE) established.
A national digital health architecture and/or health information exchange (HIE) is defined including semantic, syntactic, and organizational layers.
The HIE is operable and provides core functions, such as authentication, translation, storage, and warehousing; a guide to what data is available and how to access it; and data interpretation.
The government leads, manages, and enforces implementation of the national digital health architecture and/or the health information exchange (HIE), which are fully implemented following industry standards.
The national digital health architecture and/or health information exchange (HIE) provides core data exchange functions and is periodically reviewed and updated to meet the needs of the changing digital health architecture. There is continuous learning, innovation, and quality control. Data is actively used for national health strategic planning and budgeting.
Box 2: The Health Information Systems Interoperability Maturity Toolkit (HISIMT) categories and maturity descriptors
The HISIMT asks very specific “yes/no” questions about the various subdomains of leadership and governance, human resources, and technology to ascertain the maturity level.
Within the subdomain “human resources capacity development,” the following statements, seeking a binary answer (Yes/No), are used to calculate a level of maturity:
Nascent: The country has no training programs to build human resource capacity on digital HIS, including interoperability.
Emerging: There is a nationally-recognized preservice training curriculum outlining competencies for human resources for digital HIS and interoperability for HIS.
Established: A plan exists for in-service training of HIS staff to build their skills around digital HIS and interoperability, based on a nationally or internationally recognized HIS curriculum.
Institutionalized: The country has the capacity to train enough staff to support digital HIS and interoperability, through in-country, preservice, and in-service training institutions or partnerships with other training institutions.
Optimized: The government and its stakeholders provide sustainable resources for health ministry staff to receive training on HIS, including digital HIS and interoperability.
The Health Information System Stages of Continuous Improvement Toolkit (HISSCIT) categories and maturity descriptors:
The HISSCIT uses similar descriptors to label a maturity level as Emerging/Ad hoc, Repeatable, Defined, Managed, or Optimized
Box 3: The Informatics Capability Maturity Model (ICMM) categories and maturity descriptors
The ICMM provides general descriptors for 5 levels of maturity with the neutral maturity being standardized and standards-based.
For each of the 5 ICMM dimensions, the level of maturity may be
Basic: Unproven, disjointed, and uncoordinated processes; Knowledge not shared; Unpredictable performance. Avoiding downtime
Controlled: Knowledge silos exist; Processes manageable and getting predictable; Reactive and problem driven
Standardized: Standards; Predictable performance; Knowledge sharing within organization; Proactive but essentially request driven
Optimized: Continuous improvement; Efficiency; Cross-organization knowledge sharing and collaboration; Proactive and accountable; Service driven
Innovative: Catalyst for innovation; New dynamic processes, Industry level knowledge sharing and collaboration; Drives service innovation; Value driven
REFERENCES
- 1.Digital Health WHA71.7, Seventy-First World Health Assembly (2018). Geneva: World Health Organization. [Google Scholar]
- 2.World Health Organization. Regional Action Agenda on Harnessing E-Health for Improved Health Service Delivery in the Western Pacific. Manila: WHO Western Pacific Regional Office; 2017 [Google Scholar]
- 3.World Health Organization Global Observatory for eHealth. Global Diffusion of eHealth: making Universal Health Coverage Achievable. Report of the Third Global Survey on eHealth. Geneva: WHO; 2016. [Google Scholar]
- 4. Carvalho JV, Rocha Á, Abreu A.. Maturity models of healthcare information systems and technologies: a literature review. J Med Syst 2016; 40 (6): 131. [DOI] [PubMed] [Google Scholar]
- 5. Ansari S, Godinho M, Jonnagaddala J, et al. A systematic review of toolkits for implementation and evaluation of digital health interventions. PROSPERO CRD42019147273 2019. https://www.crd.york.ac.uk/prospero/display_record.php? ID=CRD42019147273 Accessed December 20, 2019.
- 6.World Health Organization. Global Strategy on Digital Health 2020–2024. Geneva: WHO; 2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.World Health Organization. Classification of Digital Health Interventions v1.0: A Shared Language to Describe the Uses of Digital Technology for Health. License: CC by-NC-SA 3.0 IGO. Geneva: WHO; 2018. [Google Scholar]
- 8. Hammond WE, Bailey C, Boucher P, et al. Connecting information to improve health. Health Aff 2010; 29 (2): 284–8. [DOI] [PubMed] [Google Scholar]
- 9.World Health Organization Global Observatory for eHealth. Atlas of eHealth country profiles: The use of eHealth in support of universal health coverage. Geneva: WHO; 2015. [Google Scholar]
- 10.World Health Organization & International Telecommunication Union. WHO-ITU: National eHealth Strategy Toolkit. Geneva: WHO & ITU; 2012. [Google Scholar]
- 11.World Health Organization. WHO Guideline: recommendations on Digital Interventions for Health System Strengthening. Licence: CC by-NC-SA 3.0 IGO. Geneva: World Health Organization; 2018. [PubMed] [Google Scholar]
- 12.Global Digital Health Index Consortium. Global Digital Health Index Indicator Guide. 2016. https://www.digitalhealthindex.org/ Accessed March 7, 2020. [Google Scholar]
- 13.UK Department of Health Informatics Directorate, Division of Informatics Capability Development. Informatics Capability Maturity Model (ICMM). United Kingdom: NHS; 2018. [Google Scholar]
- 14. Liaw S-T, Kearns R, Taggart J, et al. The informatics capability maturity of integrated primary care centres in Australia. Int J Med Inform 2017; 105: 89–97. [DOI] [PubMed] [Google Scholar]
- 15.MEASURE Evaluation and Health Data Collaborative. Health Information Systems Interoperability Maturity Toolkit: Users’ Guide. Chapel Hill, NC: University of North Carolina; 2017. [Google Scholar]
- 16.MEASURE Evaluation and Health Data Collaborative. Global Digital Health Resources and Maturity Models: A Summary Overview. Chapel Hill, NC: University of North Carolina; 2017. [Google Scholar]
- 17.MEASURE Evaluation and Health Data Collaborative. HIS Stages of Continuous Improvement Toolkit. Chapel Hill, NC: University of North Carolina; 2017. [Google Scholar]
- 18. Greenhalgh T, Wherton J, Papoutsi C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017; 19 (11): e367. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. May CR, Cummings A, Girling M, et al. Using normalization process Theory in feasibility studies and process evaluations of complex healthcare interventions: a systematic review. Implement Sci 2018; 13 (1): 80. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Agarwal S, Labrique A, LeFevre A, et al. Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. Geneva: World Health Organization; 2016. [Google Scholar]
- 21.World Health Organization. The MAPS Toolkit: mHealth Assessment and Planning for Scale. Geneva: WHO; 2015. [Google Scholar]
- 22. Langley J, Wolstenholme D, Cooke J.. Collective making’ as knowledge mobilisation: the contribution of participatory design in the co-creation of knowledge in healthcare. BMC Health Serv Res 2018; 18 (1): 585. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.International Telecommunications Union. Measuring the Information Society Report Volume 2 ICT Country Profiles. Geneva: ITU; 2018. [Google Scholar]
- 24.International Telecommunications Union. Measuring the Information Society Report Volume 1 2018. Geneva: ITU; 2018. [Google Scholar]
- 25.International Telecommunications Union. Measuring the Information Society Report Executive Summary. Geneva: ITU; 2018. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.