Digital maturity scores as gatekeepers for health AI: useful proxy or false comfort?

Healthcare is rapidly integrating artificial intelligence (AI). This integration promises transformative benefits for patient care and operational efficiency.¹ Many organizations use digital maturity scores. These scores assess existing technological infrastructure and capabilities. A critical question, therefore, arises: Are these scores valid proxies for AI readiness? Or do they offer a misleading sense of comfort? This article argues for a nuanced position. Digital maturity scores can offer a foundational baseline. However, they are insufficient as sole gatekeepers for health AI. AI-specific readiness assessments are fundamentally essential for safe and effective deployment. The promotion of general digital maturity models as pathways to AI adoption may inadvertently encourage an oversimplified view of complex sociotechnical requirements. Organizations understandably seek clear roadmaps for intricate technological transitions such as AI implementation.² This demand for measurable progress indicators can elevate the importance of achieving a specific maturity score. Consequently, organizations might prioritize this metric. Such a focus could overshadow the critical need for deep, AI-specific preparedness. A high general score might cultivate a superficial understanding of AI’s nuanced demands. This potentially leads to a misplaced sense of security concerning true AI readiness. Furthermore, the rapid evolution of AI, particularly generative AI, may outpace the relevance of traditional digital maturity models.³ These models were often not designed with newer AI paradigms in mind.

BACKGROUND: THE RISE OF DIGITAL MATURITY SCORES IN HEALTHCARE

Digital maturity reflects an organization’s comprehensive digital capabilities, including technology, data, governance, management, engagement, and communication.⁴ Several maturity models currently guide healthcare’s digital transformation journey. The Healthcare Information and Management Systems Society (HIMSS) offers prominent examples of such frameworks. HIMSS’s Electronic Medical Record Adoption Model (EMRAM) assesses EMR adoption levels. EMRAM features eight distinct stages, progressing from stage 0 to stage 7.⁵ This model primarily focuses on EMRs’ effective support of patients and clinicians. The HIMSS Analytics Maturity Assessment Model (AMAM) measures analytics capabilities within healthcare organizations.⁶ AMAM explicitly aims to guide these organizations toward successful AI adoption. The modernized AMAM, introduced in 2024, specifically addresses AI integration more directly.⁷

Other specialized models are also emerging. For instance, the Digital Health Communication Maturity Model (DHCMM) emphasizes user engagement and communication.⁴ This signals an evolution in thinking beyond purely technological aspects of maturity. Achieving higher digital maturity is often linked to demonstrably better organizational outcomes.⁸ Recent studies associate advanced EMRAM stages with improved patient experience and enhanced safety metrics. For example, advanced digital maturity in US hospitals correlates with stronger patient experience outcomes.⁹ This is particularly true for communication with nurses and doctors. Consistent with these findings, hospitals with advanced EMRAM stages (6 or 7) show significantly higher odds of achieving better safety grades and report reduced adverse events and improved surgical safety.

This positive association underpins a common belief that high general digital maturity might indicate an organization’s readiness for complex technologies like AI. The focus of many digital maturity models on infrastructure and EMR adoption, however, could create a pathway dependency.¹⁰ AI adoption might be viewed as a linear progression only after achieving high EMR maturity. This perception could potentially delay AI exploration in areas not heavily reliant on mature EMR systems. Some AI applications, especially those using nontraditional data sources, might be implementable with moderate EMR maturity if other AI-specific capabilities are present. The success of models like EMRAM in correlating with some positive outcomes could also paradoxically reinforce their use as a proxy for all advanced digital initiatives, including AI.¹¹ This may occur even if the specific components measured by EMRAM are not the most critical for a particular AI application.

COUNTEREVIDENCE: THE “FALSE COMFORT” ARGUMENT

Generic digital maturity scores possess inherent limitations as sole AI readiness indicators. They may not adequately capture AI-specific needs and complexities. Indeed, the overall value of digital maturity models (DMMs) is a subject of ongoing debate within the academic community.¹² A significant barrier to AI adoption identified by healthcare systems is the prevalence of immature AI tools. Among surveyed US health systems, 77% reported this concern.¹³ An organization’s high digital maturity score does not guarantee the maturity or fitness-for-purpose of the AI tools it seeks to deploy. This highlights a critical disconnect between organizational preparedness and technological readiness.

AI implementation can face substantial challenges even within digitally advanced healthcare settings. For example, sepsis prediction models are a common AI use case. These models, however, demonstrate variable performance and significant generalizability issues across different hospitals and patient populations. One study found that a single sepsis prediction model may not perform acceptably across diverse sites, even within the same multihospital system.¹⁴ Another external validation of a widely used commercial sepsis predictive model in emergency departments concluded that it failed to achieve meaningful sensitivity. The study suggested it offered little assistance to clinicians.¹⁵ This illustrates that general digital infrastructure does not equate to successful AI deployment. The complex, context-dependent nature of AI in healthcare is unlikely to be captured by standardized, generic digital maturity metrics. Such metrics might mask underlying contextual incompatibilities for specific AI tools.

Furthermore, digital maturity scores often overlook crucial prerequisites specific to AI. These include robust AI-specific data governance frameworks and ethical oversight mechanisms.¹⁶ Effective AI data governance requires attention to data quality, integrity, ethical standards, fairness, privacy, and security.¹⁵ The development of AI governance maturity, encompassing organizational structure, problem formulation, and lifecycle management, is also vital.¹⁷ Specialized AI talent and comprehensive AI literacy programs for staff are other key factors frequently unaddressed by broad maturity scores.¹⁸ The lack of internal capabilities and capacity presents a significant challenge for AI adoption.¹⁹

There is a tangible risk of these scores becoming a superficial checkbox. This can lead to “false comfort.”²⁰ Organizations might believe they are AI-ready based on achieving a high score. However, they may lack the deep, specific capabilities essential for safe and effective AI implementation. This can result in failed AI projects, wasted resources, or even unsafe deployments. The focus of some DMMs on technical infrastructure might also neglect crucial user-centered metrics. Aspects such as user satisfaction, engagement, and effective communication are vital for AI tool acceptance and practical utility.⁴ The “immature AI tools” barrier suggests a fundamental misalignment. Organizational maturity focuses on the adopter’s readiness. AI tool maturity focuses on the technology’s readiness. High organizational maturity cannot compensate for inherently flawed or underdeveloped AI. This implies a need for a dual assessment strategy, evaluating both the organization and the specific AI solution. Financial concerns also act as a significant barrier to AI adoption.¹³ Organizations may hesitate to invest in AI tools perceived as immature. Yet, without sufficient investment, these tools may not reach the necessary maturity. Digital maturity scores alone do not resolve this investment-maturity paradox for AI.

PROPOSED ALTERNATIVE: TOWARD MEANINGFUL AI READINESS ASSESSMENT

A more nuanced and comprehensive approach to AI readiness is urgently necessary. This approach must extend beyond generic digital maturity scores. Organizations should prioritize the development and assessment of AI-specific governance maturity. Digital health communication maturity models (DHCMM) provide structured, tiered governance pathways.⁴ DHCMM establishes seven maturity levels spanning initial through engaged stages. Assessment occurs across communication, engagement, personalization, and governance domains. The framework emphasizes user-centered metrics and robust governance structures. These elements facilitate AI tool acceptance and clinical implementation. Additional frameworks continue to emerge to address healthcare AI governance challenges.¹⁷

The digital competence of healthcare professionals is another critical factor. AI literacy among clinical and administrative staff significantly impacts AI adoption, effectiveness, and patient safety. Studies show that digital competence moderates the relationship between AI interventions and both operational efficiency and patient safety outcomes.¹⁸ Thus, readiness assessments must evaluate and plan for enhancing workforce AI literacy. Robust ethical frameworks must be central to any AI readiness assessment. These frameworks should guide AI development, rigorous validation, responsible deployment, and continuous ongoing monitoring¹⁶ (Figure 1). The World Health Organization, among others, provides guidance on AI ethics in health, emphasizing principles such as protecting autonomy and ensuring fairness.²¹

Figure 1.

Conceptual contrast between simplistic and multidimensional models of AI readiness. This figure contrasts the common linear view of “digital maturity score → AI readiness” with a holistic, literature-informed perspective. The multidimensional framework incorporates governance, ethics, workforce competence, data quality, model lifecycle, and workflow integration as essential components of AI preparedness in healthcare.

Assessment protocols must include a thorough evaluation of data quality and integrity, specifically for AI applications. This is particularly crucial for developing trustworthy large-language models and other data-intensive AI systems.²² An organization’s capacity for AI model validation is also essential. This includes capabilities for external validation of vendor-provided models and continuous monitoring for performance degradation or data drift post-deployment. The ability to effectively integrate AI tools into existing clinical workflows also needs careful evaluation. AI solutions must seamlessly fit into, and ideally enhance, clinical processes without causing undue disruption or increasing clinician burden.²³ The emergence of AI-specific governance models indicates that existing digital maturity frameworks cannot address AI’s unique challenges. This shift requires moving from “digital transformation” to “AI integration” readiness. The focus on digital competence and AI literacy shows that readiness depends on both human capital and technological infrastructure, impacting training and organizational culture. Integrated ethical frameworks extend AI readiness beyond technical capability to responsibility and trustworthiness.

RETHINKING AI GATEKEEPING IN HEALTH

Digital maturity scores provide a limited view of AI readiness, and relying on them alone for healthcare AI adoption is problematic. Healthcare organizations must adopt comprehensive, AI-centric readiness assessments that integrate technical, organizational, human capital, and ethical dimensions.² Balancing AI innovation with patient safety, clinical efficacy, and health equity is crucial. A holistic approach beyond simple scores will enable responsible AI adoption while minimizing risks and ensuring that technology serves patients effectively. Current gatekeeping mechanisms might exclude beneficial AI innovations or allow tools that meet superficial criteria but lack safety. Successful AI gatekeeping requires collaboration between policymakers, healthcare organizations, technology developers, ethicists, and patient advocates to implement appropriate AI readiness standards for healthcare.

Disclosure statement/Funding

The authors report no funding or conflicts of interest.