From principles to practice: an actionable framework for AI governance in healthcare organisations

Abstract

Objective

Artificial intelligence (AI) technologies are being rapidly adopted in healthcare, yet organisational governance often lacks the processes needed to oversee their safe and responsible use. Previous AI governance frameworks largely focus on high-level AI ethics principles, leaving healthcare organisations struggling to translate these principles into practice, assess risk and embed AI oversight into existing processes. This study aimed to develop and validate a practice-oriented AI governance framework for healthcare organisations.

Methods and analysis

We developed an AI governance framework using an exploratory multimethod design that drew on a scoping review, document analysis and semistructured interviews. The framework was validated through stakeholder workshops and application to three case studies of AI systems to identify patients for home-based care, radiology decision support and infection surveillance within a large tertiary healthcare organisation. The case studies represented real-world governance decisions faced by healthcare organisations and reflected the major pathways for acquiring AI capabilities: in-house development, commercial procurement and codevelopment with external partners.

Results

The framework unifies ethics and governance principles, incorporates tiered oversight aligned with organisational digital maturity and includes a structured review checklist to support consistent decision-making about the implementation and operation of AI systems across clinical and operational contexts. Stakeholders found the framework relevant, usable and feasible to implement. Across the diverse AI use cases, the review checklist identified governance, safety and implementation risks and supported structured assessment and decision-making. The framework also enabled AI oversight to be integrated into existing organisational governance processes while supporting a proportionate approach to risk management across the AI life cycle.

Conclusion

This study addresses an important gap between high-level AI governance principles and organisational implementation. The framework provides healthcare organisations with a structured approach to assessing, governing and monitoring AI systems throughout their life cycle and may support the safe and responsible adoption of AI in practice. Further research is needed to evaluate the framework across diverse healthcare settings and emerging AI technologies, including generative and agentic AI.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Existing artificial intelligence (AI) governance frameworks primarily focus on ethics principles and provide limited guidance on how healthcare organisations can operationalise AI governance within established safety, quality and digital health governance structures.

WHAT THIS STUDY ADDS

• This study presents an AI governance framework that combines tiered oversight with a structured review process to support risk assessment, accountability and life cycle governance of AI systems within healthcare organisations.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• The framework provides a structured approach for integrating AI governance into existing organisational processes and may inform future implementation, evaluation and policy development for the governance of AI in healthcare.

Introduction

Artificial intelligence (AI) technologies are being rapidly adopted in healthcare organisations across clinical and non-clinical tasks, from predicting patient care needs and supporting diagnosis to optimising administrative processes.¹ Yet, their integration into healthcare also raises persistent ethics and safety concerns, including bias in decision-making, unequal access to benefits, lack of transparency and risks from errors due to data drift.^2,3 Despite rapid progress, many organisations still lack the governance structures and decision-making processes to identify and manage these risks effectively.⁴ Concerns around transparency, accountability, privacy and fairness can overshadow the potential benefits of AI, particularly when patient harm can occur at scale or go undetected.⁵ Generative AI systems such as ambient scribes, virtual assistants and discharge planners are already being used in clinical settings, despite unresolved issues around accuracy, accountability and workforce impact.⁶ In this evolving context, healthcare organisations increasingly face a growing need for AI governance or formal processes to ensure the safe and responsible use of AI technologies.^5,7

Even as demand grows, there is limited guidance on governing the safe and responsible development, implementation and use of AI.⁷ Although many theoretical frameworks exist, most provide broad ethics and governance principles, leaving a gap between principle and practical application.⁸ Few frameworks in the current literature have been tested or implemented in healthcare settings, and even fewer are aligned with the complex workflows and decision-making processes of healthcare organisations. It also remains unclear how ethics and governance responsibilities should be operationalised across distinct stages of the AI life cycle or embedded into existing processes for patient safety and digital health.⁹ Additionally, limited attention has been given to the varying levels of digital maturity needed to assess, implement and monitor AI effectively across diverse healthcare contexts.⁵

This study addresses the gap between high-level AI ethics principles and their practical implementation in healthcare. To close this gap, we developed and validated an actionable governance framework to support healthcare organisations to adopt and use AI systems safely and responsibly. We use the term ‘AI systems’ to refer broadly to AI-enabled models, tools, applications, software, platforms and workflow-integrated systems used for clinical, operational, administrative or research purposes. The framework is intended as a principles-based organisational governance framework and should be applied proportionately according to the AI system’s intended use, risk, autonomy, complexity, data access, clinical impact and integration into organisational workflows. As it was primarily developed and tested in the context of non-agentic AI systems, the framework will require further development and testing before it is fully applicable to agentic systems.

Methods

Study design and setting

We used an exploratory multimethod approach¹⁰ to develop and test a governance framework for the safe and responsible use of AI in healthcare organisations. The study was designed in accordance with the Consolidated Criteria for Reporting Qualitative Research (COREQ).¹¹ It comprised four stages as outlined in the published protocol¹²: (1) understanding AI governance needs through a document analysis and a scoping review of the literature; (2) stakeholder interviews; (3) developing a draft framework through synthesis of findings; and (4) validating and refining the framework through stakeholder workshops and by application to AI case studies involving systems that were in use or under consideration at the study site (figure 1). Data were analysed using NVivo V.15 (2024).

Figure 1. Study process diagram. AI, artificial intelligence.

The study was conducted at a major public healthcare organisation in Australia serving a catchment of over 770 000 people in Melbourne’s inner south. The organisation operates across three hospital campuses, including a tertiary and quaternary referral hospital providing specialised care. Its additional campuses focus on community-based care including general medicine, geriatric medicine, rehabilitation and mental health. It also delivers state-wide services across the state of Victoria (eg, Victorian Adults Burns Service) and national services across Australia (eg, paediatric lung transplant service).¹³ As a large, complex and digitally enabled health service, this setting provided a robust environment for developing and testing a practice-oriented AI governance framework for real-world care delivery settings. Data collection took place between April and July 2024.

Patient and public involvement

This study was informed by prior deliberative public engagement, including an Australian citizens’ jury on the use of AI in healthcare, which developed nationally representative recommendations on when and how AI should be used and governed in health systems.¹⁴ The jury’s findings highlighted the need for strong organisational governance, independent oversight, patient rights and clear accountability and helped shape the study’s focus on operational governance mechanisms. These insights were complemented by consultation with hospital-based consumer groups during the research, which informed consideration of ethics, safety and acceptability issues (see stage 2). Thus, patient and public involvement in the development of the framework was both consultative and indirect, rather than following a codesign process. Patients and the public were not involved in data analysis or manuscript preparation.

Participants

Participants were recruited for the semistructured interviews (stage 2) and stakeholder workshops (stage 4). Individuals with professional experience relevant to the design, implementation or oversight of healthcare AI systems were eligible. These included healthcare professionals, researchers, data and analytics specialists, legal and ethics experts, governance leads, executives, board members and health consumer representatives.

For the interviews, two key informant groups were targeted: (1) healthcare staff and consumers from the study site and (2) subject matter experts from academia (ethics, governance, digital health, technology, AI), government, healthcare administration, clinical settings and professional healthcare associations. A purposive sampling strategy was used to identify individuals meeting the inclusion criteria, followed by snowball sampling to allow participants to refer additional relevant stakeholders. Healthcare staff were recruited in consultation with the study site’s project team and selected based on their experience in domains such as clinical operations, digital health, data governance, legal, ethics, research and executive leadership. Subject matter experts were identified through the scoping review,¹⁵ referrals and professional networks. Interviews were conducted with participants from Australia, New Zealand, USA, UK and Hong Kong. Deidentified characteristics of the subject matter expert interview participants are provided in online supplemental file 4.

Stakeholder workshop participants were recruited at the study site. The first workshop included a multidisciplinary cohort with diverse institutional representation, while the second involved members of the board of directors. All workshop participants were identified and invited in consultation with the study site’s project team. Written informed consent was obtained from all participants prior to participation.

Procedures

The framework was developed and refined in four stages.

Stage 1: understanding governance needs

An initial structure for the framework was developed and refined based on the results of the scoping review, which is reported separately.¹⁵ The scoping review examined 77 frameworks for healthcare organisations implementing AI systems for clinical or operational purposes. From these, we derived four components that form a practical AI governance framework:

1. Set of guiding principles.

2. Organisational oversight mechanism (eg, governance committee).

3. Method to identify, review or assess AI systems (eg, checklist items, supporting materials).

4. Consideration of governance processes across the AI life cycle stages.

We identified the theoretical and practical components of AI governance, synthesising ethics and governance principles, oversight mechanisms and checklist elements drawn from best practices, operational guidance and ethics standards relevant to acute care. As the framework developed, the AI life cycle stages (component 4) were absorbed into the review method; this was to streamline the process and ensure each assessment item was grounded in a specific stage of the AI life cycle.

To ensure alignment with existing organisational governance, a document analysis was conducted at the study site. This involved a comprehensive review of internal policies, governance documents and procedural guidelines relevant to digital health and patient safety. The objective was to establish a baseline of current processes for oversight, identify strengths and gaps and avoid duplication in the proposed AI governance framework.

Stage 2: gathering insights on AI governance

43 semistructured interviews were conducted with two key informant groups: 23 with healthcare staff and 20 with subject matter experts. Thematic saturation was reached for both groups. Additional input was sought through consultations with two hospital-based consumer reference groups, one focused on digital health and the other on diversity and inclusion. These consultations informed the refinement of governance priorities, particularly around acceptability, transparency, safety, equity and representation.

Interview guides were developed based on a literature review and input from the multidisciplinary project team, with experience in healthcare, qualitative research, AI safety and governance. No interview guide, prompts or question schedule was provided, and no pilot testing was undertaken. Participants had no prior relationship with the interviewer or research team and were provided no personal information about the researcher beyond the participant information and consent form, which outlined the purpose and rationale for the study. Participants were approached via email, and interviews were conducted during standard work hours. Interviews took place either in person or via videoconferencing and were audio recorded. No repeat interviews were conducted, and transcripts were not returned to participants for comment or correction. Interviews lasted approximately 30–60 min.

Recordings were deidentified and professionally transcribed using a secure transcription service. Two researchers independently coded the transcripts using grounded theory and open coding techniques (SF, AW). A grounded theory approach enabled the inductive development of concepts that were grounded in the experience and professional expertise of participants. This method was selected because it provides a systematic procedure for generating theory from empirical data where existing models may be limited or underdeveloped.¹⁶ The analysis informed the framework by identifying practical governance needs, common challenges and priority areas for safe and responsible AI in healthcare. Key themes extracted from the interviews included data governance, ethics and legal accountability, workforce capability, risk stratification, clinical integration and equity. These findings shaped the structure and content of the ethics and governance principles, oversight mechanisms and review checklist components. Findings from the 20 subject matter expert interviews are reported separately.¹⁷

Stage 3: developing the draft framework

Findings from the scoping review,¹⁵ document analysis and interviews were synthesised to develop a draft AI governance framework tailored to the operational context of healthcare delivery organisations. The draft framework incorporated regulatory requirements, international best practices and stakeholder insights to support applicability. From the 77 frameworks identified in the scoping review,¹⁵ 10 of these that contained an available checklist were selected to inform the development of the initial AI review checklist, a component of the framework. They were selected because they provided practical questions, reporting items, review criteria or assessment prompts that could be adapted. The checklist was then refined using the document analysis, stakeholder interviews, workshops and case study testing, with mapping against the ethics and governance principles from the 77 frameworks used to assess coverage and identify areas requiring further refinement. The ethics and governance categories identified across the 77 included frameworks were coded and mapped (online supplemental file 3). These categories were used as an input to the subsequent synthesis and refinement of the final principle set and checklist, together with findings from the document analysis, stakeholder interviews, workshops and case study testing. During this process, source categories were retained, renamed, consolidated or refined, and additional principles were incorporated where supported by the wider synthesis. The final principles are defined in online supplemental file 1, appendix 5. Checklist questions relating to agentic AI were also added following review of an additional agentic AI governance framework, which highlights agent-specific considerations including autonomy, tool and system access, human approval points, agent identity and permissions, technical controls, logging, monitoring and multi-agent risks.¹⁸ The draft framework also outlined integration pathways with existing clinical and operational processes. It was refined in stage 4 through stakeholder workshops and tested and refined by applying it to three AI case studies.

Stage 4: testing and refining the framework

Structured workshops were conducted with internal stakeholders to validate and refine the draft governance framework. Participants were identified in consultation with the study site (n=23). Workshops followed a guided facilitation approach, incorporating structured discussion and scenario-based evaluation to assess feasibility, usability and alignment of the framework with existing governance structures and processes. Feedback was thematically analysed and used to iteratively revise the framework. The draft framework, with a focus on the AI review checklist, was then tested and refined using three case studies of AI systems in use or under consideration at the study site. These comprised:

1. Home-based Eligibility Analysis & Recommendation Tool

Summary: this in-house developed AI tool was created to help identify hospital patients who may be suitable for home-based care. The tool supports an existing process by which clinicians identify patients who may be suitable based on certain criteria. It reviews patient records two times a day and suggests those most likely to meet eligibility criteria. This helps clinical teams make faster decisions, improves patient outcomes and frees up hospital beds. However, staff remain in control, with the tool supporting not replacing clinical judgement.

2. Aidoc CT tool¹⁹

Summary: this commercially developed AI tool is used in radiology to help flag urgent findings on CT scans, such as brain bleeds, spine fractures or blood clots, so radiologists can review them faster. The goal is to reduce delays, improve patient safety and support quicker treatment decisions. Radiologists remain responsible for reviewing scans and all related decisions, with the AI tool serving as an additional review to help detect serious issues earlier.

3. Clostridioides difficile classification tool

Summary: this codeveloped (university and hospital) AI tool helps infection prevention teams to classify C. difficile infections. It uses hospital data to automatically apply national definitions, saving time and reducing the risk of manual errors. The tool supports surveillance and reporting, but final decisions rest with the clinical team.

These case studies were selected because they were either operational or under active consideration at the study site and represented real-world governance decisions faced by a healthcare delivery organisation. They reflected the different pathways through which healthcare delivery organisations acquire AI capabilities including in-house tool development, procurement of commercial AI products or via codevelopment in partnership with external organisations. They were not intended to represent all possible AI system types and did not include generative, agentic, multi-agent or highly autonomous AI systems.

For each case study, project teams completed the checklist using a think-aloud protocol.²⁰ This allowed real-time feedback on the usability, relevance and clarity of the checklist items. Feedback was analysed thematically and used to revise checklist content, clarify definitions and improve logical flow. The case studies tested the framework’s applicability across selected clinical, operational and research-related governance contexts at the study site and demonstrated its flexibility and utility in surfacing risks, clarifying roles and informing decision-making. Findings from this stage, combined with earlier insights, informed the final version of the governance framework.

Results

Practice-oriented AI governance framework for healthcare organisations

The AI governance framework presented here provides a structured, organisational-level approach to the safe and responsible integration of AI in healthcare. It comprises three components: (1) a set of ethics and governance principles; (2) oversight mechanism (based on digital maturity); and (3) a review checklist to support consistent decision-making about the implementation and operation of AI systems across clinical and operational contexts. In the following sections, we present each component, first describing it and then outlining its application at the study site.

Ethics and governance principles

We identified 30 ethics and governance principles for healthcare organisations to consider when implementing AI (see online supplemental file 1, appendix 5). Each organisation will need to review and prioritise these principles based on their local context, risk profile and existing governance structures.

At the study site, the first step involved mapping their existing organisational principles, followed by aligning each one with corresponding checklist questions (see online supplemental file 1, appendix 4; online supplemental file 2). This process supported the operationalisation of the principles within the framework and ensured that they were addressed either through the checklist or through existing governance mechanisms. It also ensured that the principles were not treated as abstract elements but embedded into decision-making, implementation and evaluation.

During the workshop (stage 4), stakeholders reviewed a consolidated list of ethics principles and prioritised those that were either poorly represented in existing policies or that posed new risks in the context of AI. These included accountability, consent, cultural appropriateness, fairness, transparency, sustainability and trust. Some, such as dignity and well-being, were considered to already align well with existing organisational principles. Participant discussions highlighted the challenge of distinguishing between principles already embedded in governance processes and those requiring AI-specific adoption. Participants stressed the need to differentiate what can be managed at the organisational level (eg, privacy) from broader societal risks (eg, regulation). The process was considered valuable in understanding these considerations and identifying practical pathways for operationalising high-level principles within an AI governance framework tailored to a local context.

Oversight mechanism

Three dominant mechanisms for organisational oversight emerged from our analysis. These included: (1) creating a dedicated AI governance committee; (2) expanding an existing governance committee to include AI oversight; and (3) integrating AI governance into existing non-AI-specific governance processes. Recognising that the suitability of these mechanisms depends on an organisation’s digital maturity and available resources, we incorporated them into a tiered approach to organisational oversight.

Digitally mature organisations

For digitally mature organisations with adequate infrastructure and workforce capabilities, a dedicated AI governance committee was suggested (figure 2). This high-level, multidisciplinary body would provide centralised oversight of AI evaluation, approval and monitoring, supported by a broader advisory group and an operations group responsible for implementation. The advisory group provides subject matter expertise as required for specific AI systems and makes recommendations to the governance committee. The operations group supports applicants to complete the checklist, conducts risk stratification for AI systems, responds to queries and provides the administrative functions that enable the governance committee and advisory group to fulfil their oversight roles (online supplemental file 1, appendix 2). The advisory group is shown with dashed lines in figure 2 to indicate that it provides advice as required. An AI use register can also be established (online supplemental file 1, appendix 6) as part of this oversight function to promote transparency and accountability and to support ongoing monitoring, evaluation and decommissioning of AI systems.

Figure 2. Artificial intelligence (AI) oversight in a digitally mature healthcare organisation through a dedicated AI governance committee.

Developing digital capacity organisations

For healthcare organisations with developing digital capacity, AI oversight can be delegated to an existing multidisciplinary team, such as a digital health or clinical innovations committee, with clearly defined roles and domain-specific advisors (figure 3). This committee would use the AI review checklist to guide evaluations and provide formal recommendations to an existing governance body, such as an executive or digital health committee. An AI advisory group could also be established to support the existing governance committee.

Figure 3. Artificial intelligence (AI) oversight in a healthcare organisation with developing digital capacity by expanding an existing multidisciplinary committee.

Limited digital capacity organisations

For healthcare organisations with limited digital capacity, where both infrastructure and workforce capabilities are constrained, AI governance can be initially embedded within existing oversight mechanisms such as clinical governance, digital health, ethics and procurement processes (figure 4). A gap analysis can then be conducted to identify any unaddressed responsibilities, with the AI review checklist serving as the central evaluation tool. This approach would support practical, distributed governance by leveraging existing resources, while maintaining alignment with safety, ethics and regulatory requirements. In these settings, the checklist should be used to support approval decisions, as well as to identify gaps in available expertise and determine when internal or external specialist input is required.

Figure 4. AI oversight in a healthcare organisation with limited digital capacity by integrating governance into existing clinical governance, digital health, ethics and procurement processes. AI, artificial intelligence; IT, information technology.

Stakeholders noted that healthcare organisations vary widely in their digital capability, which influences both the scope of AI they can safely adopt and the governance structures they can sustain. For organisations with limited digital capacity, embedding AI oversight within existing processes offered a pragmatic starting point. These organisations were viewed as often having constrained infrastructure and limited specialist expertise and were best positioned to focus on targeted AI use cases that were operationally critical or lower in risk. This approach enabled proportionate oversight while avoiding the significant administrative and workforce investment required to govern more complex AI systems. It also recognised that, at early stages of maturity, organisations may not yet be equipped to make consistent and well-informed decisions about higher risk AI applications.

In contrast, organisations with more mature digital capability were seen as being better equipped to support a broader and more proactive approach to AI governance. These services typically had stable infrastructure, defined digital governance systems and access to specialist expertise across data governance, privacy, cybersecurity, clinical governance and informatics. As they transitioned to a more embedded governance model, a dedicated AI advisory or governance committee could continue to set organisational priorities, assess strategic fit and provide structured oversight of safety, ethics and implementation. Mature organisations were also viewed as having the resources required to ensure that AI systems progressed efficiently through approval pathways, were monitored appropriately and were integrated into existing governance functions over time. This was viewed as enabling a longer term framework in which AI governance became embedded across executive and operational roles, supported by an executive-level function guiding prioritisation and organisational alignment.

AI oversight mechanism at the study site

At the study site, internal stakeholders supported establishing a high-level, multidisciplinary AI governance committee during the early stages of AI integration. This committee would provide centralised oversight for ethics, clinical, operational, legal and technical considerations, with a phased plan to embed responsibilities within existing structures over time. A direct reporting line to the executive committee (the main body exercising managerial responsibility for the performance of all activities within the organisation) was favoured to ensure visibility, organisational alignment and resourcing. Two alternative options, reporting to the digital health steering committee (accountable for overseeing and setting the overall strategic direction across all digital health programmes of work) or directly to the board (effectively bypassing the executive committee), were considered less effective due to perceived gaps in focus or proximity to organisation-wide operational decision-making. While a dedicated AI governance committee was considered suitable in the early stages of adoption, it was recognised that over time these responsibilities should transition into existing governance and executive functions as organisational capability, confidence and experience in reviewing AI systems increase.

To support implementation, a smaller ‘AI operations group’ was proposed. This group would provide technical and logistical support, assist with completion of the AI review checklist and manage the day-to-day coordination of AI governance activities. A broader advisory group of subject matter experts (eg, clinical, legal, cybersecurity, consumer representatives, etc) would be engaged per use case.

Stakeholders noted that many existing governance committees lacked the necessary breadth of expertise to manage AI-related risks, particularly in data science, ethics and emerging and evolving regulation. Our governance framework aims to address this gap by supporting AI governance literacy, creating structured approval pathways and enabling robust monitoring. To support this, organisations are recommended to develop tailored internal policies and terms of reference that clearly define the governance committee’s scope, functions and decision-making authority. The committee’s proposed functions, reporting lines and membership serve as a practical foundation for governing clinical and operational AI systems within a care delivery setting (see online supplemental file 1, appendix 1).

To support proportionate oversight and avoid unnecessary administrative burden, several participants recommended introducing a preliminary screening or ‘gating’ process. This approach enables organisations to distinguish between high and low-risk AI activities based on a small number of objective criteria. For example, a project may be referred for formal AI governance review if it involves identifiable data, requires integration into hospital information technology (IT) infrastructure or requires organisational resources such as ongoing IT support. Tools that do not meet these criteria can generally proceed through existing research or quality improvement processes. Integrating a short gating step into AI intake processes can help ensure that governance attention is directed to areas of greatest organisational risk and impact (see online supplemental file 1, appendix 3).

AI review checklist

A central component of the framework is the AI review checklist, a structured evaluation tool designed to guide consistent, transparent and evidence-informed oversight of AI systems across clinical and operational domains. It supports both research and operational pathways and aims to ensure that the use of AI aligns with ethics standards, organisational priorities and regulatory requirements, while being adaptable and flexible in its application. The checklist was developed through the synthesis of AI governance and implementation frameworks that contained a checklist,^21–30 identified through the scoping review¹⁵ and additional agentic AI governance framework,¹⁸ and key domains from the informant interviews and document analysis. Each question was mapped to ensure broad coverage of key checklist items (online supplemental file 1, appendix 3). Checklist items were iteratively refined based on informant interviews, mapping of ethics and governance principles (online supplemental file 2) and the AI case studies.

The final checklist comprises 10 domains and includes items relevant to clinical, research and operational applications of AI (table 1). Of these, seven reflect key stages in the AI life cycle to support comprehensive oversight from concept and design to development, deployment, operational use, review, and retirement or replacement (figure 5). Further rationale for domain inclusion is provided in online supplemental file 5. The life cycle is used to structure governance review so that relevant risks, responsibilities and evidence requirements are considered at the appropriate stage.³¹ The remaining three domains relate to initial screening, background details and administrative/funding considerations. These were also included as supporting governance domains rather than life cycle stages, because they provide information needed to determine accountability, feasibility, resourcing and organisational ownership across the life cycle (table 1).

Table 1. AI governance checklist core domains.

Domain	Purpose
1. Screening intake	Identifies risk level, life cycle stage, pathway, data use, integration requirements, system type and whether enhanced review may be required.
2. Background details	Captures organisational, project, vendor and contact information.
3. Concept	Defines the problem, rationale, value proposition, evidence base and strategic alignment.
4. Design	Describes intended use, model type, outputs, stakeholder involvement, consumer considerations, governance requirements and data/outcome definitions.
5. Development	Covers data preparation, model development, performance, validation, privacy, cybersecurity and technical assurance.
6. Deployment	Reviews implementation readiness, workflow impact, user training, escalation, redress, continuity planning and human–AI interaction.
7. Operational	Specifies monitoring, maintenance, safety triggers, change management and learning health system alignment.
8. Review	Assesses post-deployment evaluation, equity, performance changes, data quality issues and actual versus expected outcomes.
9. Retirement or replacement	Covers decommissioning, replacement, records management, workflow impacts and lessons learnt.
10. Administrative/funding	Captures funding, implementation costs, technical support and endorsement.

AI, artificial intelligence.

Figure 5. Artificial intelligence (AI) life cycle stages.

While the checklist (online supplemental file 1, appendix 3) was designed to be comprehensive, it is not intended to function as a mandatory full-form assessment for every AI system. Rather, it should be applied as a modular review framework, with the depth and scope of review determined by the AI system’s risk profile, intended use, complexity, autonomy, data requirements, clinical or operational impact and organisational context. Lower risk systems may require only a basic review or confirmation that relevant issues are already addressed through existing processes, while higher risk, patient-facing, generative, agentic or deeply integrated systems may require targeted or full review with specialist input. Application of the checklist to generative and agentic AI systems remains provisional, as these system types were not included in the testing undertaken in this study. This flexible, context-specific use of the checklist is intended to support proportionate governance and integration into existing workflows without introducing unnecessary duplication or burden. In this context, practical implementation requires organisations to define local risk thresholds, review pathways and specialist input requirements, so that checklist completion is proportionate to governance maturity, available expertise, organisational risk tolerance and the specific risks of the AI system under review.

In organisations with developing or limited digital capacity, the checklist may also function as a gap identification tool, helping to determine when additional internal or external expertise is required rather than expecting the reviewing body to resolve all technical, legal, clinical, privacy or cybersecurity issues itself. Each item in the checklist is linked to relevant ethics and governance principles (eg, transparency, fairness, accountability), ensuring these concepts are embedded in the assessment process rather than treated as separate considerations. Applicants are asked to complete only the questions relevant to their proposed pathway (research, clinical/operational or both), with clear instructions and subheadings for readability.

Feedback from the three case studies led to refinements including clearer definitions, improved logic flow and the ability to reference existing documentation rather than duplicating content. The checklist now serves as both a practical tool for governance and an educational mechanism to build AI literacy among developers and decision-makers (see online supplemental file 1, appendix 3 for complete AI review checklist). A completed worked example of the AI review checklist is provided to illustrate how the checklist can be applied to an AI use case (online supplemental file 6). It is intended to support structured deliberation and documentation of governance decisions rather than operate as a numerical scoring instrument or automated approval tool. A rationale for each checklist domain, including the governance risks each domain is intended to address, is provided in online supplemental file 5.

Discussion

This study describes the design, testing and refinement of a practical AI governance framework tailored to the needs of healthcare delivery organisations. The framework addresses a persistent gap between high-level AI ethics and governance principles and the real-world processes required to govern the safe and responsible use of AI in healthcare. It includes a mapped set of ethics and governance principles, possible committee structures with defined oversight and a practical checklist for reviewing AI systems aligned with key stages in the AI life cycle. By combining a comprehensive scoping review with stakeholder input and real-world testing, the framework connects normative and procedural approaches to AI governance, supporting translation into practice. As it was primarily developed and tested in the context of non-agentic AI systems, the framework will require further development and testing before it is fully applicable to agentic systems.

Our findings align with previous research that has characterised AI governance as fragmented, heavily principle based and largely lacking in operational tools.^32,33 While many current frameworks present high-level concepts such as transparency, fairness and accountability, few provide a clear path for applying these concepts in healthcare delivery environments. The limited uptake of governance tools may reflect this disconnect, as well as uncertainty among healthcare leaders about where governance responsibility lies. Prior work has highlighted the challenge of operationalising governance in health systems, particularly when governance expectations are vague or insufficiently contextualised. Pyone et al, in their systematic review of health systems governance, highlighted many governance frameworks that lacked practical guidance for implementation at different levels of the health system.³⁴ Similarly, Brinkerhoff and Bossert emphasised the disparity between formal governance structures and the ‘extent to which these are actually put into practice’.³⁵

This study responds to those concerns by defining a structured governance framework with aligned tools and processes that can be integrated into existing organisational processes. Accounting for AI life cycle stages (figure 5) enables continuous oversight throughout the development, implementation and use of AI systems. Each stage is supported by relevant principles, ensuring that governance is both continuous and proportional to risk. This life cycle-based approach supports iterative, context-specific governance models that respond to the evolving nature of AI use in clinical and operational settings.^36,37

Monitoring requirements should be defined during approval and adapted to the AI system’s intended use, risk profile, baseline performance, deployment context and acceptable outcomes. Time-based review assesses whether the AI system remains appropriate within a changing deployment context, while event-triggered review responds to specific incidents, updates, safety signals, complaints, performance degradation or other predefined escalation criteria. Time-based review also provides an additional assurance mechanism when relevant event-based triggers may not be fully known at the time of approval, helping to identify emerging risks, workflow drift, changes in data quality, shifts in user behaviour or unintended consequences that may not yet generate a predefined incident or escalation trigger.

As technology continues to evolve, some AI systems may require enhanced review because of their technical or operational characteristics. Currently, these include generative AI systems, agentic AI systems, systems that can access tools, Application Programming Interfaces (APIs) or clinical records, systems that act on behalf of users, systems with autonomous or semiautonomous decision-making and systems where vendor-controlled updates may alter performance or behaviour after approval. For these systems, governance review requires additional assessment of permissions, delegated authority, human approval points, audit logging, reversibility, escalation processes, runtime monitoring, and shutdown or rollback mechanisms.

Our stakeholder consultations confirmed that current governance structures do not routinely account for AI-specific considerations. Participants reported that project approval pathways vary across clinical, research and digital domains, with little clarity around who is responsible for assessing AI-related risks. These insights support those of Morley et al and Jobin et al, who have highlighted the challenges of operationalising ethics principles across complex healthcare systems.^38,39 Key gaps identified by stakeholders include insufficient attention to bias mitigation, post-deployment safety, integration with clinical workflows and the ability to contest or override AI-generated outputs. The review checklist was developed to address these needs and was tested on three AI systems representing different acquisition pathways, with relevant stakeholders, including frontline and operational users who assessed its usability and relevance within real-world organisational conditions. This testing process demonstrated the checklist’s flexibility across different acquisition pathways (internal development, commercial procurement and codevelopment) and its ability to elicit detailed, structured discussion about model purpose, performance, implementation challenges and ongoing monitoring. Unlike more conceptual self-assessment tools, this checklist directly supports project-level review and approval. Participants found it useful both as a decision-making aid and as a tool to build AI literacy within multidisciplinary teams.

The framework includes defined roles and committee structures that align with existing healthcare governance models. The AI governance committee oversight should function similarly to that of quality, safety and innovation governance committees. This design reflects a deliberate strategy to embed AI oversight within existing organisational processes, reducing duplication and ensuring alignment with broader service priorities. Here, the framework aims to promote both integration and validity. Additionally, the framework is intended to build on existing clinical safety, privacy, cybersecurity, procurement, ethics, regulatory or risk management processes. Many requirements are not unique to AI but require coordinated application because AI systems often cut across these domains. The framework therefore intends to function as a coordination layer, supporting the identification of which existing processes apply, where AI-specific evidence or interpretation is needed and where additional review, monitoring or escalation is required.

As organisations gain experience with AI and build capability to assess and manage associated risks, there is growing support for AI governance responsibilities to transition from stand-alone committees into existing executive and governance functions. With appropriate upskilling, staff across domains such as clinical governance, research ethics, data governance, privacy and cybersecurity will be able to incorporate AI considerations within their usual decision-making processes. An executive role, such as a chief AI officer or equivalent, could oversee the prioritisation, need and suitability of AI systems, working alongside other executive functions to ensure consistent, efficient and accountable decision-making across the organisation.

The framework also addresses equity and inclusion through multiple mechanisms. The inclusion of cultural appropriateness, fairness and transparency as core principles was prioritised by stakeholders, particularly in relation to diverse data governance and health equity considerations. The framework recommends consulting relevant Indigenous, First Nations, minority group, culturally and linguistically diverse and vulnerable populations (eg, children), including Aboriginal and/or Torres Strait Islander advisors where relevant to the Australian context. This consultation should be considered during project planning, recognising that ethics risks and opportunities are not evenly distributed across populations. This aligns with strength-based, community-led approaches to digital health governance.⁴⁰ Additionally, consumer and public involvement should be proportionate to the AI system’s function, risk profile, patient-facing nature and equity implications, with deeper involvement required where systems directly affect patient communication, access, prioritisation or care decisions.

While the framework is grounded in the Australian health system, its components are designed to be adaptable across diverse healthcare contexts. Its development draws on widely accepted ethics and governance principles, and its tools can be localised to reflect different regulatory requirements and institutional structures and processes. The modular design of the checklist and committee structure allows for phased implementation or adaptation to resource-constrained settings. This flexibility is essential, given the rapidly evolving nature of AI technologies and the varied levels of digital maturity across healthcare organisations.⁴¹ Risk-tiering thresholds and review pathways should be refined over time as organisations gain experience reviewing AI systems, identify common risk patterns and build local governance capability. The AI governance framework and associated tools are provided in online supplemental file 1. To support implementation and local adaptation, an up-to-date online version is also available at: https://www.mq.edu.au/faculty-of-medicine-health-and-human-sciences/departments-and-schools/australian-institute-of-health-innovation/healthcare-tools-and-resources.

Several limitations warrant discussion. The framework was developed and tested within a single healthcare organisation and will likely require modification in other contexts. Although the checklist was validated on three AI use cases, a larger sample across multiple settings would strengthen generalisability. Additionally, validation was limited to AI systems that were operational or under active consideration at the study site and did not include generative, agentic, multi-agent or highly autonomous systems. These technologies may pose additional governance and evaluation challenges that were not examined in the present validation, including variable outputs across repeated runs, a lack of fixed reference standards for evaluating some free-text outputs, version-dependent changes in performance or behaviour, vendor-controlled updates and the burden on clinicians of verifying generated outputs. As their use in healthcare increases, further work is needed to assess whether the expanded tiering model and enhanced review processes adequately address these challenges. Longitudinal evaluation of the framework’s implementation would also be valuable, including its effects on governance decision-making, project outcomes and staff engagement. Finally, future research should explore governance models in which AI is not only the object of review but also supports governance activities, such as intake, documentation, monitoring, implementation support and human-in-the-loop review.

Notwithstanding these limitations, this study provides one of the first practice-oriented frameworks for AI governance in healthcare that has been developed and validated within a real-world healthcare organisation. The framework is designed to be adapted by other healthcare organisations seeking to enhance their governance processes to ensure the safe and responsible use of AI. It contributes a practical approach to the emerging field of digital health governance, grounded in systems thinking, real-world constraints and inclusive design. As organisations continue to explore the potential of AI, frameworks such as this one will be essential to guide adoption and manage risk. AI governance must move beyond principles and into systems, processes and shared accountabilities. This study contributes an applied governance approach that can support transformation in healthcare without compromising safety or responsibility.

Data availability statement

Data are available upon reasonable request.