Scaling enterprise AI in healthcare: the role of governance in risk mitigation frameworks

Abstract

This perspective article examines the role of governance frameworks in mitigating risks and building trust in AI implementations within healthcare organizations. As AI technologies rapidly evolve, robust governance is essential to manage potential adverse incidents and ensure fair, equitable, and effective innovation. This article highlights key risks associated with AI deployments and proposes enhancements to enterprise AI governance to better address these challenges posed by AI and digital health innovations.

Introduction

One of the most pressing challenges facing healthcare today is a severe workforce shortage, particularly in areas such as nursing, primary care, and psychiatry, causing concerns about patient access and quality of care due to a lack of available healthcare professionals. The American Hospital Association estimates that the industry will face a shortage of up to 124,000 physicians by 2033¹. Meanwhile, it will need to hire at least 200,000 nurses a year to meet rising demands¹. AI-powered solutions offer the potential to alleviate this strain by automating administrative tasks, supporting precision diagnostics, and enhancing personalized care. For instance, state-of-the-art generative AI-enabled chatbots can assist patients with administrative tasks during clinic check-ins, while more traditional machine learning algorithms can analyze medical images to flag abnormalities that could otherwise be missed during clinical image interpretation by a radiologist.

Despite the promise of AI, its deployment in healthcare also carries significant risks. These include concerns about data privacy, algorithmic bias, transparency, and potential unintended hallucinations with impact on patient safety². To ensure that AI is used responsibly and ethically, it is imperative to establish robust governance frameworks that mitigate these risks across the AI supply chain. Such AI governance frameworks can become the springboard for safe and effective innovation in healthcare settings.

The adoption of AI governance in healthcare is increasingly being shaped by federal agencies that have introduced guidelines to promote safety, effectiveness, and transparency. Agencies like the Office of the National Coordinator for Health IT (ONC) and the Food and Drug Administration (FDA) have issued regulations for AI-enabled medical devices and certified health IT^3,4, reinforcing governance as a foundation for building trustworthy AI. At the same time, many AI applications in healthcare—particularly those used for administrative tasks—fall outside formal regulatory oversight and instead depend on self-governance by developers and users. As both public and private organizations race to establish frameworks for responsible AI use, enterprise leaders are often left navigating a crowded and sometimes conflicting landscape of standards, best practices, and evolving expectations.

This article aims to explore the core tenets of enterprise governance for effective AI risk mitigation in healthcare settings. By examining key risks introduced by AI, reviewing current regulatory requirements, and drawing on industry expertise, we will propose enhancements to existing enterprise governance frameworks that can better protect stakeholders and ensure the responsible development and deployment of machine learning and generative AI-powered healthcare solutions.

Background: AI in healthcare and the evolution of governance

Overview of AI and its applications in healthcare

Artificial intelligence (AI) is transforming healthcare by improving patient care, boosting operational efficiency, and helping address challenges such as workforce shortages. AI technologies—ranging from predictive analytics that anticipate patient outcomes to machine learning and generative AI tools that support diagnostics and treatment planning—are becoming integral parts of everyday clinical and administrative workflows.

While recent public awareness of AI has been influenced by generative AI applications like OpenAI’s ChatGPT⁵ and Google’s Gemini^6,7, the use of AI in healthcare predates these developments. Earlier forms of AI, such as expert rules-based systems, have long been used in healthcare. Healthcare chatbots built as expert rules-based systems have been used for a range of applications, including patient triage, appointment scheduling, medication reminders, and chronic disease management. They also support mental health, provide insurance and billing assistance, and help clinicians with documentation and decision support. These tools proven to improve efficiency, enhance patient engagement, and expand access to timely care are now seeing a renaissance with the advent of generative AI⁸.

Healthcare institutions—payers, and healthcare IT developers—are actively integrating generative AI into their operations. Mayo Clinic, for instance, is working with Google Cloud to use generative AI for improving clinical documentation and patient communication⁹. Elevance Health (formerly Anthem) is developing its own generative AI tools to personalize member engagement and streamline claims processing¹⁰. Similarly, Optum, a subsidiary of UnitedHealth Group, is using large language models to automate prior authorizations and summarize complex patient data, showcasing how generative AI is becoming a core component of healthcare AI strategies¹¹.

To effectively integrate and utilize these advanced technologies, a robust governance framework is essential. Such a framework helps healthcare enterprises navigate the complexities of AI adoption, implementation, and production quality assurance by providing structured guidance on best practices and regulatory compliance. A robust governance framework enables organizations to make informed decisions about AI procurement and ongoing maintenance, ultimately helping unlock new commercial opportunities and enhancing patient outcomes through high-fidelity AI solutions.

Historical context of governance and risk management in healthcare technology

The integration of technology into healthcare workflows has always necessitated robust governance and risk management practices. Historically, the adoption of new technologies, such as electronic health records, medical devices, and telemedicine, was accompanied by the development of governance frameworks that were designed to ensure patient safety, data security, and regulatory compliance. These frameworks have evolved to address the unique challenges posed by technological advancements necessary often to respond to regulatory demands such as safeguarding patient privacy under the Health Insurance Portability and Accountability Act; ensuring the safety and efficacy of medical devices regulated by FDA; and, most recently, supporting the adoption of trustworthy AI in healthcare settings mandated by the Office of the National Coordinator Health Data, Technology, and Interoperability (ONC HTI-1) rule. ONC HTI-1 Rule Section (b)(11) requires transparency for AI and predictive algorithms in certified health information technology (IT) systems. The rule mandates that health IT developers must disclose clear information about how their AI algorithms function, including their development process, data sources used for training, and known limitations. This transparency enables healthcare providers to properly evaluate the risks associated with using these AI tools, understand their limitations, and make informed decisions about implementing them in clinical settings. Essentially, ONC HTI-1 (b)(11) ensures that AI systems in healthcare are not “black boxes”—clinicians must be able to understand how the technology arrives at its recommendations or conclusions.

The current governance landscape for AI in healthcare is shaped by a combination of federal and state regulations, as well as industry-specific standards. At the federal level, agencies such as the Department of Health and Human Services, through the Office of Civil Rights (OCR), the Office of the National Coordinator for Health Information Technology (ONC) and FDA, play pivotal roles in setting regulatory requirements for AI in healthcare. These agencies have begun to establish guidelines, standards, and frameworks to address the unique challenges posed by AI, including ensuring the safety, effectiveness, and ethical use of AI-driven medical devices and health IT systems.

State-level regulations also contribute to the governance of AI in healthcare, with some states like Utah¹² and Colorado¹³ enacting their own laws and guidelines to address data privacy, security, and the ethical implications of AI use. Additionally, industry organizations like the Coalition for Healthcare AI (CHAI) and international standards bodies like the International Organization for Standardization (ISO)¹⁴ are developing best practices and guidelines to support the responsible adoption of AI in high-risk settings like healthcare. These bodies have established standards for AI system scorecards, quality validation across diverse populations, and AI management system requirements through frameworks like ISO 42001. Common principles across these guidelines include ensuring patient privacy and data security, maintaining clinical workflow integration, regular performance auditing, establishing healthcare professional training requirements, and implementing clear adverse event reporting processes. These evolving frameworks aim to balance AI innovation with necessary safety measures while addressing healthcare’s unique regulatory and ethical challenges.

Governance of AI for healthcare

Before we dive deeper into introducing the core pillars of a robust AI governance framework for healthcare enterprises, we want to propose a definition for AI governance. Specifically, AI governance refers to the frameworks, policies, and processes that guide the ethical and responsible design, development, procurement, deployment, and use of artificial intelligence. In healthcare, where AI technologies are increasingly integrated into administrative and clinical decision-making, effective governance can ensure AI systems get decommissioned before they become a safety hazard. By managing risks such as bias, privacy concerns, and hallucinations, AI governance helps build trust in these technologies and ensures they contribute positively to healthcare outcomes.

Governance is the bedrock of trust in AI for healthcare enterprises and without strong governance, AI systems can easily become sources of harm rather than benefit. For example, in the case of AI-powered diagnostic tools, governance ensures that the data used to train these tools is diverse, appropriate for the purpose at hand, and free from bias, preventing inaccurate diagnoses that could disproportionately affect certain patient groups. Additionally, AI governance mandates regular audits of AI algorithms to identify and mitigate any unintended consequences, such as a predictive model unfairly prioritizing healthcare access for some patients over others. By establishing clear fairness standards, transparency, and accountability measures, governance reassures end users, clinicians, and regulators that AI innovations are safe, equitable, and aligned with the overarching mission of improving healthcare outcomes. This trust is crucial for the successful integration and acceptance of AI in healthcare settings.

AI risks in healthcare settings

Risks in healthcare AI applications can emerge at various stages, from data collection to algorithm implementation. One key area of concern is data quality. AI models rely heavily on the data they are trained on, and if this data is incomplete, biased, or unrepresentative, the resulting predictions or recommendations can be flawed. Similarly, AI models rely on high-quality data inputs in real-world settings, and if the data they receive is incomplete or noisy, their predictions can become inaccurate or unreliable. For instance, an AI system trained primarily on data from a pediatric population may perform poorly when applied to a more diverse patient population, leading to misdiagnosis or inappropriate treatment recommendations. Moreover, if an AI system is used to determine the medical necessity of a procedure based on poorly documented medical records, it may produce inaccurate results, potentially delaying or denying patients the care they need.

Algorithmic bias is another significant risk in AI applications. Even when trained on high-quality data, AI systems can unintentionally reinforce existing disparities if the underlying training data reflects systemic inequities. This risk is especially challenging when healthcare organizations adopt AI tools from external vendors, as they often lack visibility into how the training data was selected or governed. Without transparency, it’s difficult to assess whether the AI model is representative or fair. For instance, a risk assessment tool might overestimate the likelihood of certain conditions in specific ethnic groups due to biased data, leading to inappropriate treatment decisions.

Interpretability is another challenge in the governance of AI applications in healthcare. Complex AI models, like generative AI that is built on deep learning algorithms, often operate as “black boxes,” making it difficult for healthcare providers to understand how specific decisions are made. This lack of understanding can erode trust and hinder the adoption of AI solutions in healthcare settings where interpretability and explainability are essential for informed decision-making.

Finally, implementation risks emerge when AI systems are integrated into healthcare workflows without sufficient planning and oversight. Issues such as poor data quality or limited data availability can significantly impact the performance of AI-powered solutions in real-world settings. Additionally, without clear implementation protocols, there is a risk that clinicians may misuse these tools or rely too heavily on their recommendations. For instance, an AI system designed to optimize surgical schedules could unintentionally prioritize billing efficiency over clinical urgency, ultimately compromising patient care.

Addressing these risks requires robust governance frameworks that ensure AI systems are developed, tested, implemented and maintained with a focus on fairness, transparency, and accountability.

Strengthening existing governance frameworks

Traditional enterprise governance frameworks—originally built to manage general operations and compliance—must now evolve to address the unique challenges of AI, such as data quality, algorithmic bias, interpretability, and implementation risks.

1. Ensuring AI Suitability through Rigorous Problem Definition: Before any AI solution is pursued, governance frameworks must require a structured assessment of the underlying problem and a clear justification for why AI is the appropriate tool to solve it. Many failures in AI implementation stem from organizations treating AI as a catch-all solution, without fully exploring simpler or more effective alternatives. This step involves defining the business or clinical problem in measurable terms, evaluating the feasibility of non-AI solutions, and confirming that AI brings unique value compared to traditional methods. Embedding this practice in governance protocols helps avoid misguided investments and ensures that AI is only used where it is truly fit for purpose.

2. Integrating AI-Specific Data Governance: Healthcare enterprises should develop or enhance their data governance policies to include objective standards for data used in AI systems. This includes ensuring data completeness, representativeness, and freedom from bias during both training and production. Existing data governance frameworks—such as those used in clinical research or protocol development—can be expanded to support continuous monitoring of data quality, with mechanisms to detect and correct issues like incomplete, noisy, or skewed data. Additionally, as generative AI becomes more integrated into healthcare workflows, governance must account for the risk of data poisoning introduced by AI-generated content being fed back into systems. Incorporating safeguards to validate the integrity and provenance of all inputs, including AI outputs, is essential to maintaining reliable and trustworthy AI performance.

3. Enhancing Transparency and Technical Disclosures: AI governance must also address the challenge of transparency and user disclosures. Enterprises can strengthen their governance frameworks by requiring comprehensive documentation of AI model development, including data lineage, feature selection, validation methods, and bias mitigation strategies. While vendor scorecards from AI registries can offer helpful summaries of a model’s performance and compliance posture, they are not a substitute for internal evaluation. Healthcare organizations must independently validate the model’s performance on their own data to ensure that vendor-reported results hold true in their specific clinical or operational context. This internal validation is essential for identifying hidden risks or performance degradation that may not have been apparent in the vendor’s testing. Additionally, a cross-functional AI governance committee—including data scientists, clinicians, compliance officers, and ethics experts—should be established to review this data and make decisions throughout the entire AI lifecycle. This committee ensures that AI systems are rigorously assessed, monitored, and adjusted as needed, both before and after deployment.

4. Consolidate Enterprise Compliance Reporting: Healthcare enterprises are under growing pressure to comply with regulations surrounding AI systems, which requires a comprehensive approach to compliance reporting. Centralizing and consolidating compliance reporting is crucial for streamlining oversight processes and ensuring that all relevant data and incident reports are gathered in one place. By creating a unified framework that integrates compliance requirements and assigns clear accountability, healthcare organizations can improve accuracy, reduce redundancies, and ensure more efficient oversight. This centralized system enables real-time monitoring of AI implementations, allowing organizations to quickly address regulatory updates or unexpected incidents. Additionally, a consolidated reporting approach fosters transparency and accountability, which builds trust with regulators, patients, and partners. Ultimately, by consolidating compliance reporting, healthcare enterprises can operate more efficiently, stay in line with evolving standards, and ensure both patient safety and ethical AI usage.

5. Strengthening Implementation Oversight: Finally, the integration of AI systems into healthcare workflows demands robust oversight to mitigate implementation risks. Enterprises should revise their governance frameworks to include continuous monitoring and post-deployment auditing of AI systems. This could involve establishing AI usage protocols that prevent over-reliance on AI recommendations, ensuring that human oversight remains a key component of every operational workflow. For instance, governance policies could mandate that AI-driven decisions, such as those optimizing surgical schedules, are reviewed by clinicians to ensure that patient needs remain a priority.

Ongoing oversight over AI implementations must focus on the monitoring of predefined Key Performance Indicators (KPIs). Such performance metrics can be defined during the initial evaluation of the AI product and historically logged to track potential model degradation. While some KPIs can be tracked automatically with existing enterprise software monitoring tools, metrics like AI model fairness might require manual red-teaming exercises to be conducted by specialized professionals. Red-teaming for AI systems involves simulating adversarial attacks to identify vulnerabilities, assess risks, and improve the security and robustness of the AI model. Finally, it is common to expect the enterprise audit team to develop a new department or contract externally specialized resources to periodically conduct manual audits of the AI model performance in production.

By systematically augmenting existing governance frameworks with these AI-specific considerations, healthcare enterprises can better manage the risks associated with AI technologies and build a foundation of trust that supports safe, effective, and ethical AI adoption.

Extending governance to generative AI and large language models: governance and risk management

Generative AI and large language models (LLMs), such as GPT-4, present powerful opportunities to improve healthcare delivery, patient engagement, and operational efficiency. However, their use also introduces significant technical and ethical risks—ranging from factual inaccuracies to embedded social biases. Effective AI governance must therefore integrate both technical safeguards and ethical oversight to ensure responsible deployment of generative AI. This includes strategies like red-teaming foundation models, conducting ongoing ethical reviews, and updating model training processes. By framing AI governance around these pillars, healthcare organizations can proactively identify risks, mitigate bias, and uphold patient trust while driving innovation with generative AI.

A key risk management strategy for generative AI applications is red-teaming, which involves testing the AI model under adversarial conditions to uncover potential vulnerabilities, biased outputs, or inaccuracies across diverse demographic groups. This process ensures that AI applications are safe, fair, and reliable before deployment in clinical settings. Additionally, many healthcare organizations build domain-specific applications using open-source foundation models, enabling rapid innovation but also introducing hidden risks. If the base model is not rigorously evaluated, these applications may inadvertently perpetuate biases, such as unequal triage recommendations or access to services. To prevent these outcomes, a comprehensive validation framework that includes red-teaming is needed to assess both the technical accuracy and societal impact of the AI models.

In parallel, ongoing ethical review processes are essential for ensuring that generative AI applications align with healthcare values and patient-centered care. These reviews should draw input from multidisciplinary stakeholders on the AI governance commtitee—including clinicians, ethicists, data scientists, and community advocates—to surface blind spots and guide value-sensitive design decisions. Ethical review should not be a one-time activity but an ongoing feedback loop that continuously monitors and assesses the evolving impact of the AI model in real-world settings.

Lastly, adaptive model training and fine-tuning are essential for addressing emerging risks and meeting evolving societal expectations. When risks are identified, organizations should update the training data to improve representation or real-world scenarios during model training and adjust the AI algorithm design to enhance its fair performance. These technical adjustments, guided by ethical insights, ensure that generative AI continues to evolve in ways that promote equitable healthcare outcomes.

Conclusion: a path forward for AI governance in healthcare

The healthcare industry is at a critical juncture in redefining the relationship between technology and human health, aiming to ensure that the benefits of AI are accessible to everyone, everywhere. By mitigating key risks in AI implementations, developing flexible AI governance frameworks, and fostering collaboration among functional stakeholders, healthcare enterprises can create a safer and more equitable healthcare environment. AI governance committees should play a central role in reviewing and monitoring AI implementations, ensuring that AI systems meet enterprise standards for safety, efficacy, and ethical considerations. Together, we can champion responsible and ethical AI usage that not only improves patient care but also reshapes the future of healthcare for the better.

Author contributions

A.B. and J.T. contributed equally towards writing the main manuscript text.

Data availability

No datasets were generated or analysed during the current study.

Competing interests

The authors declare no competing interests.