Barrier check study: why predictive machine learning struggles to reach the operating room

Abstract

Objectives

Machine learning (ML) has the potential to enhance surgical decision-making through real-time risk prediction and personalised care yet clinical implementation remains limited. This study aimed to identify and categorise the key barriers to implementing ML tools in surgical practice.

Methods

A nationwide qualitative survey was conducted among stakeholders involved in ML development or clinical implementation in healthcare. Participants provided open-ended responses describing perceived barriers, their impact and potential solutions. Responses were analysed using inductive thematic content analysis.

Results

95 participants from 53 organisations submitted 178 individual barrier entries. The most frequently reported barriers included limited clinician trust and artificial intelligence literacy, challenges integrating ML into surgical workflows, regulatory misalignment, insufficient data availability and quality, and unclear economic value. Barriers were highly interconnected, with challenges in one domain often reinforcing others. Proposed solutions emphasised interdisciplinary collaboration, early end-user involvement, improved transparency and training, privacy-preserving data-sharing approaches such as federated learning, and better alignment between regulatory, institutional and clinical structures.

Conclusion

The findings indicate that ML implementation in surgery is a socio-technical challenge rather than a purely technical one. Barriers often arise from misalignment between responsibility, decision authority and accountability across technical, clinical and organisational domains, particularly in time-critical surgical settings.

Introduction

The integration of machine learning (ML) into surgical practice has emerged as a promising innovation, enabling real-time risk prediction and personalised decision-making based on large volumes of patient data.^{1 2}

Despite these promising applications, the implementation of ML in surgery faces numerous barriers.³ These include technical challenges such as model reliability, real-time deployment and integration with existing surgical workflows, as well as ethical concerns surrounding algorithmic bias, transparency and accountability. Moreover, regulatory hurdles and the need for clinical validation further complicate the adoption of ML in routine surgical practice.⁴

Literature reviews consistently highlight that a critical first step toward the successful integration of ML in healthcare would be clear definition and understanding of these implementation barriers.³ Identifying the most relevant obstacles in day-to-day clinical practice enables the development of targeted strategies that align with the practical needs of healthcare providers and patients.⁵

While prior studies have examined barriers to artificial intelligence (AI) and ML implementation across healthcare more broadly, these analyses have largely focused on diagnostic, outpatient or administrative contexts.⁶,⁹ Surgical practice presents distinct constraints, including time-critical intraoperative decision-making, irreversible interventions that may uniquely influence ML adoption and integration. To date, these surgical-specific considerations have not been systematically examined.

This study was designed to identify these barriers for ML implementation in surgery using a holistic, interdisciplinary approach.

Methods

This study employed a qualitative survey to examine perceived barriers to the implementation of ML in surgery. Participants provided open-ended responses identifying barriers, categorising them, rating their impact and suggesting potential solutions. They were allowed to submit multiple barriers. Informed consent was obtained from all participants prior to data collection.

This study is not considered medical research under Dutch law. It involved no medical interventions, no patients and no use of patient data, only opinions from healthcare professionals and experts. Therefore, ethical approval was not required under the Dutch Medical Research Involving Human Subjects Act.

Participant criteria

Study participants were required to have an institutional affiliation (1) and be involved in at least one ML project or in the decision-making process of an ML project (2). Additionally, all levels of experience were eligible to participate in the study. Potential participants are excluded if they do not have direct involvement in the healthcare sector (1).

Nationwide survey

A nationwide survey was conducted targeting specialists with experience in or association with AI-related projects in healthcare. The survey, administered using Qualtrics, focused on identifying perceived barriers and facilitators to the clinical implementation of ML tools in surgery. The questionnaire was developed by a team of AI researchers in close collaboration with practising surgeons. The full survey instrument is provided in the online supplemental appendix.

For the purpose of this study, predictive ML refers to models trained on historical clinical data to estimate the likelihood of future clinical outcomes or events, including applications such as risk stratification, prognostic modelling, clinical decision support and outcome prediction to inform perioperative planning and treatment decisions.

Participants were initially recruited via the networks of the research team and affiliated stakeholders within academic medical centres. Additionally, a public call for participation was distributed through social media platforms to reach a broader audience. Interested individuals who met the inclusion criteria were invited to contact the lead author (SBH) to receive a private link to the survey. Inclusion continued until data saturation was achieved.

Saturation analysis

To assess data saturation, the structured approach developed by Guest et al was followed.¹⁰ Analysis began with the 10th questionnaire round per category, with each subsequent 10 barriers examined for the emergence of new themes. Saturation was defined in terms of thematic novelty as fewer than 5% of identified themes being novel. This threshold was used to monitor saturation rather than as a stopping criterion, and data collection continued to ensure representation across participants and perspectives until additional barriers no longer contributed meaningful thematic variation.

Data analysis

Survey responses were imported into ATLAS.ti (V.23.4.0) to support data organisation and facilitate the coding process.¹¹ The data were coded and analysed using inductive thematic content analysis, with an initial clustering of conceptually similar barrier statements used to generate preliminary codes. These codes were iteratively refined and abstracted into higher-level themes across clusters, enabling identification and interpretation of key themes related to technical, operational, ethical and regulatory barriers.

Discrepancies between the two independent coders (SBH, CG) were identified through comparison of coded segments and were primarily related to differences in the interpretation or level of abstraction of specific barrier statements (eg, whether a barrier was best categorised as technical, organisational or regulatory). These discrepancies were resolved through iterative discussion and refinement of code definitions until consensus was reached in all cases.

Results

Respondent demographics

A total of 144 survey invitations were distributed, and 95 individuals responded, yielding a response rate of approximately 66%. Collectively, the respondents submitted a total of 178 individual barrier entries. As shown in figure 1, survey participants represented a range of professional groups. The largest group was senior researchers (n=36), followed by medical professionals (n=28). Other groups included respondents from industry (n=11), junior researchers (n=9), research support roles (n=8) and stakeholders or patient advocates (n=3).

Figure 1. Respondents were categorised into six overarching professional groups: senior researchers, medical professionals, industry professionals, junior researchers, research support and stakeholder/advocate roles.

Further breakdowns of each group are shown in figure 2. Senior researchers most commonly worked in development, ethics and law. Medical professionals primarily specialised in surgery (n=11) and radiology (n=7), with smaller numbers across other specialties including oncology, intensive care and anaesthesiology. Industry respondents were typically engaged in development, consultancy or data science. Junior researchers and those in research support roles covered various fields such as clinical informatics, shared decision-making and education. Stakeholder roles included patient advocacy and radiology oversight.

Figure 2. (A) Fields represented by senior researchers (n=36). (B) Medical specialties of respondents identifying as medical professionals (n=28). (C) Occupational roles among industry professionals (n=11). (D) Roles of junior researchers (n=9). (E) Functions represented within research support (n=8). (F) Stakeholder and advocacy roles (n=3).

Academic levels are summarised in the online supplemental appendix. The most common levels were full professor (n=24), PhD (n=23) and MSc (n=19), followed by PhD candidates (n=9), associate professors (n=8), MDs (n=5), assistant professors (n=2) and one EngD.

Respondent affiliations

Survey respondents represented a range of institutions across academia, healthcare, industry and the public sector. In total, affiliations were reported by individuals from 53 distinct organisations.

As shown in figure 3, the most frequently represented institution was Amsterdam UMC (n=23), followed by Erasmus MC (n=5), Maastricht University (n=5), Vrije Universiteit Amsterdam (n=4), University of Amsterdam (n=4) and UMCG (n=4). Other institutions with multiple respondents included TU Delft (n=3), Maastricht UMC+ (n=3), Radboudumc, UMC Utrecht, SAS and Pacmed (each n=2). Three participants did not report an affiliation. All other affiliations were represented by a single respondent.

Figure 3. Word cloud of reported affiliations among survey respondents (n=95). Font size corresponds to the number of respondents associated with each institution.

Saturation analysis

The saturation threshold of fewer than 5% newly emerging themes was first reached at the 19th barrier, corresponding to the 13th participant. Subsequent barriers contributed additional depth and reflected variation in participant perspectives rather than substantively new themes, with minor fluctuations in novelty observed until approximately the 60th barrier. Beyond this point, no meaningful thematic variation across participants was observed, and data collection continued through the 178th barrier (95th participant). A saturation curve showing the rolling percentage of new themes across barrier responses is displayed in the online supplemental appendix.

Quantitative insights

The distribution of the 178 reported barriers across barrier clusters is summarised in figure 4. Circle size indicates the total number of barriers per cluster, while internal segments reflect contributions from different professional sectors. The distribution shows that barriers were concentrated in a limited number of clusters. The most frequently reported barriers were related to data (n=29), trust (n=24), Medical Device Regulation (MDR) (n=21) and commerce (n=20). A second group of commonly reported barriers included workflow (n=16), decision support (n=13) and education (n=11). All remaining barrier clusters were reported less frequently, with several clusters receiving fewer than five total entries.

Figure 4. Distribution of reported barrier clusters across professional sectors. Circle size represents the total number of barriers reported within each class (n=178). Segments within each circle indicate the contribution of different professional sectors.

Stratification of the same barrier cluster distribution by respondent role is shown in the online supplemental appendix, distinguishing surgeons, other medical professionals and non-medical professionals. Across nearly all barrier clusters, the majority of reported barriers originated from non-medical professionals. This pattern was particularly pronounced for data, trust, MDR and workflow, where non-medical professionals accounted for the largest share of entries. Surgeons contributed barriers across most major clusters but represented a smaller proportion of total reports, most notably within data, trust and commerce. Other medical professionals contributed across a similar range of clusters, with visible representation in workflow, commerce and education. In smaller barrier clusters (eg, ethics, usability, culture change and sustainability), reports were sparse and predominantly contributed by non-medical professionals.

Qualitative insights

Human factors: trust, literacy and skill retention

Trust-related concerns were raised in 24 responses. Clinicians were described as hesitant to adopt ML tools due to a lack of perceived transparency, unclear performance boundaries and general scepticism toward predictive models in high-stakes environments.

As one respondent stated:

Surgeons lack trust in the technology. Would they allow the ML model to make mistakes? — Industry Data scientist

AI education was mentioned in 11 responses. Participants emphasised that many healthcare professionals lack the training to evaluate or interpret model outputs effectively. Some also pointed to the risk of deskilling and automation bias, the idea that clinicians might come to depend too heavily on ML, eroding independent clinical judgement.

Proposed solutions included increasing AI literacy through targeted training programmes, incorporating explainability into model design and involving clinicians early in the development process. One participant emphasised the importance of making ML systems interpretable:

Use model explainability to show how the model works. Also, report on the results of working with and without the model. — Industry Data scientist

Others proposed that the role of ML should be carefully positioned within the clinical workflow so that opacity becomes less of a limitation.

Workflow integration and clinical decision support

Sixteen responses highlighted difficulties in integrating predictive ML into established clinical workflows, particularly in the operating room. Respondents pointed out the misalignment between ML tools and existing routines, as well as the lack of support from institutional information technology (IT) departments.

One respondent noted:

More proactive and less risk-averse IT support is needed to enable workflow integration. — Medical professional Radiology

Solutions emphasised early end-user involvement (n=4) and user training (n=4) as essential for uptake. Others discussed the need to clearly define how predictive tools should inform decisions. In three responses, the lack of a structured ‘decision frame’ was highlighted as a key barrier, reflecting an absence of shared understanding on how to act on the model’s outputs. Suggested remedies included structured consensus approaches such as Delphi studies.

Additionally, one response addressed human oversight as a barrier, and proposed dedicated research into oversight mechanisms for ML-supported decisions.

Regulatory and institutional alignment

Regulatory compliance challenges were mentioned in 21 responses, particularly in the context of the EU’s MDR and the proposed AI Act. These included general concerns about legal requirements, delays in certification and questions around liability, especially for models that are continuously updated.

One participant captured the tension between innovation and regulation:

Training algorithms continuously from new data is not possible but updating ‘static’ algorithms based on research/modelling on the new data is. But it requires a controlled process compliant to the MDR. — C-Suite Executive Industry

Several respondents stressed the need to involve legal expertise early in the development process (n=7), while others proposed adapting current regulations to better accommodate dynamic models (n=2).

Structural solutions were also proposed. One respondent suggested:

A one-stop proactive station/role within the organization … that provides actionable knowledge and ideally takes part of the responsibility. — Senior researcher

This theme also included responses (n=6) about broader institutional barriers, such as misaligned use cases, overly narrow model designs and lack of collaboration between AI developers and clinical teams. Respondents called for stronger alignment with healthcare needs, coordinated strategy and leadership support.

Data availability, quality and Infrastructure

Data-related barriers were raised in 28 responses. The most common concern was data quality (n=14), followed by limited availability (n=8) and access (n=6). Some respondents also pointed to data drift (n=2), warning that shifts in real-world data over time could degrade model performance.

In six responses, electronic health record (EHR) systems were identified as a core challenge and opportunity. Participants emphasised the need for closer collaboration with EHR providers such as HIX and Epic to improve data integration. Suggestions included establishing ‘middle-layer’ connection rules and technical infrastructure to ensure consistent, secure and meaningful data exchange between predictive ML models and clinical systems.

Bias in data and model performance was raised in eight responses. Participants emphasised the importance of using diverse datasets and conducting multicentre validation to ensure generalisability. In addition, some called for structured ethical deliberation to determine what trade-offs in fairness and performance are acceptable in clinical settings.

As one respondent explained:

Robust AI models require large, realistic datasets, yet access remains challenging. Small, clean clinical study datasets fail to produce real-world-ready models… large-scale databases for AI research are still scarce. — Senior researcher

Solutions included fostering institutional collaboration to improve data access (n=5), standardising data formats (n=4) and implementing governance frameworks (n=4). Three participants also recommended federated learning as a privacy-preserving alternative to centralised data sharing.

Commercial and economic feasibility

Twenty responses addressed barriers related to the commercial implementation of ML. A key concern was the difficulty in demonstrating economic value (n=5), which in turn made it hard to secure institutional support or funding. Some noted that healthcare institutions lacked incentives to adopt ML tools, particularly when benefits were indirect or long-term.

Proposed solutions included identifying inefficiencies ML could address (n=4), involving funders and insurers early in the process (n=3) and engaging key stakeholders to ensure that development was aligned with practical goals and financial realities. One respondent emphasised:

We need incentive structures that reward implementation, not just academic publication. — Industry Data scientist

Funding limitations were mentioned in three responses, and a few participants called for more active engagement with health insurers and national funders (eg, NWO) to support implementation efforts.

An overview of the barriers and their associated solutions is given in figure 5.

Figure 5. Overview of barriers and solutions for clinical implementation of predictive machine learning. Stakeholder responses highlight interconnected challenges spanning human factors (trust, AI literacy and deskilling), workflow integration, regulatory and institutional alignment, data quality and infrastructure, and commercial feasibility. AI, artificial intelligence.

Discussion

The barriers identified in this study should not be interpreted as a comprehensive restatement of known challenges in AI implementation but rather as an indication of where implementation efforts in surgical ML most consistently stall. Rather than reiterating individual barrier categories, the findings point toward a core insight: predictive ML implementation in surgery fails when responsibility, decision authority and accountability are not explicitly aligned across technical, clinical and institutional domains. This misalignment is particularly consequential in surgical contexts, where decisions are time-critical, responsibility is highly individualised and opportunities for corrective feedback are limited.

From an implementation science perspective, these findings underscore that surgical ML adoption is a socio-technical challenge rather than a purely technological one. Consistent with frameworks such as NASSS (nonadoption, abandonment, scale-up, spread and sustainability), barriers arise from interactions between adopters, organisations and the wider system rather than from isolated domains such as technology readiness or regulation.¹²

While many of these barriers have been reported in broader AI-in-healthcare studies, our findings highlight how they manifest in surgical contexts shaped by critical decision-making, individualised responsibility and the need for clear decision frames and tight workflow integration.⁶,⁹ As these challenges persist across respondent groups, incremental technical improvements alone are unlikely to enable sustainable adoption without parallel institutional and governance adaptations.

An additional observation is that most barriers originated from non-medical professionals. This may reflect limited clinician exposure to technical and regulatory aspects of ML or a difference in perspective, where non-clinical stakeholders focus more on infrastructural and governance challenges while clinicians prioritise more practical workflow and usability challenges.

Importantly, the findings suggest that collaboration is necessary but insufficient if it remains informal or ad hoc. Respondents repeatedly pointed to the absence of clearly defined structures for collaboration, decision-making and oversight as a root cause underlying multiple barriers. This shifts the implementation challenge from one of awareness or willingness to one of organisational design. Framing ML implementation as a wicked problem is therefore useful insofar as it directs attention toward iterative governance, role clarity and adaptive institutional mechanisms rather than one-off solutions or linear deployment pathways.

These interpretive insights also point toward several practical directions for institutions and stakeholders seeking to translate predictive ML into surgical practice. Rather than broad calls for collaboration or alignment, respondents implicitly described a need for concrete organisational and governance mechanisms.

One recurring implication is the value of formalised collaboration and institutional structures to oversee ML implementation. Establishing dedicated AI governance bodies, bringing together surgical leadership, data science expertise, legal and regulatory specialists, IT and hospital management may help align use cases, clarify responsibility and integrate regulatory considerations early in the development lifecycle. Such structures can move implementation efforts away from isolated pilot projects toward coordinated institutional strategies, as illustrated by existing collaborative initiatives such as REAiHL, the Erasmus MC Data Hub and the LUMC CAIRELab, which provide institutional platforms for joint clinical, technical and governance-oriented AI development.¹³,¹⁵

The findings further suggest that clinician readiness should be operationalised rather than assumed. Instead of general appeals to ‘improving AI literacy’, institutions and developers alike may consider defining minimum competencies focused on interpreting model outputs, understanding uncertainty and recognising appropriate decision boundaries. This need is reinforced by regulatory developments, as Article 4 of the EU AI Act explicitly obliges providers and deployers of AI systems to ensure an adequate level of AI literacy among users.¹⁶ Educational efforts that are co-designed with surgeons, OR assistants and embedded within existing training or quality-improvement activities may foster trust while mitigating risks such as automation bias or deskilling.

Finally, early and sustained end-user involvement emerges as a practical necessity rather than a best-practice ideal. Embedding surgeons and perioperative teams throughout model development, validation and workflow integration can help ensure that outputs align with real decision points and time constraints in the operating theatre. Structured consensus-building approaches, such as Delphi methods or multidisciplinary design workshops, may support the development of shared decision frames for ML-supported care.

Strengths and limitations

A key strength of this study lies in its interdisciplinary participant pool, which included clinicians, researchers, industry professionals and stakeholders involved in different stages of ML development and implementation. This breadth of perspectives allowed the analysis to capture barriers beyond technical performance alone, encompassing organisational, regulatory, ethical and economic dimensions that are often underrepresented in clinically focused studies. The use of open-ended qualitative responses further enabled participants to articulate context-specific challenges and proposed solutions in their own terms, supporting thematic depth and saturation.

Several limitations should be considered when interpreting these findings. First, the respondent sample was geographically concentrated in the Netherlands, with a substantial proportion affiliated with Amsterdam UMC, a large academic medical centre with established digital infrastructure and active involvement in AI development. This institutional concentration may have influenced the types of barriers reported. For example, respondents operating within a well-resourced academic environment may be more attuned to regulatory, governance and data stewardship challenges, while barriers related to basic digital infrastructure, data availability or IT capacity may be underrepresented compared with less academic settings.

In addition, the prominence of regulatory frameworks such as the Medical Device Regulation and the proposed AI Act reflects the European policy context in which many respondents operate. As a result, the relative importance of specific barriers may differ in healthcare systems with alternative regulatory, reimbursement and liability structures. For example, in mixed public-private systems such as the United States, barriers related to reimbursement, procurement incentives and malpractice liability may play a more central role than the regulatory alignment challenges emphasised in this European context.

Despite these contextual differences, several of the identified barriers appear to be transferable across healthcare systems. In particular, challenges related to trust, workflow integration, data quality and the need for clear decision frameworks were consistently reported across respondent groups and are likely inherent to the socio-technical nature of ML implementation in high-stakes clinical environments.

Conclusion

The findings of this study highlight that successful implementation of ML in surgical settings requires addressing more than technical performance. Cultural, institutional, regulatory and data-related factors all play significant roles in shaping acceptance and integration. Trust, transparency, clinician involvement and alignment with existing workflows and legal structures emerged as central themes. These insights underscore the need for a coordinated, multidisciplinary approach that involves clinicians, technologists, legal experts and healthcare administrators from the earliest stages of development. Addressing these barriers holistically may increase the likelihood of meaningful and sustainable ML adoption in surgical practice.

Data availability statement

No data are available.