Artificial intelligence for the public sector: opportunities and challenges of cross-sector collaboration

Abstract

Public sector organizations are increasingly interested in using data science and artificial intelligence capabilities to deliver policy and generate efficiencies in high-uncertainty environments. The long-term success of data science and artificial intelligence (AI) in the public sector relies on effectively embedding it into delivery solutions for policy implementation. However, governments cannot do this integration of AI into public service delivery on their own. The UK Government Industrial Strategy is clear that delivering on the AI grand challenge requires collaboration between universities and the public and private sectors. This cross-sectoral collaborative approach is the norm in applied AI centres of excellence around the world. Despite their popularity, cross-sector collaborations entail serious management challenges that hinder their success. In this article we discuss the opportunities for and challenges of AI for the public sector. Finally, we propose a series of strategies to successfully manage these cross-sectoral collaborations.

This article is part of a discussion meeting issue ‘The growing ubiquity of algorithms in society: implications, impacts and innovations’.

1. Introduction

An ambition to be the world's most innovative economy is set out in the UK Government Industrial Strategy. The Artificial Intelligence (AI) Sector Deal institutionalizes the partnership between government, industry and academia in achieving this key ambition by aiming to attract and retain domestic and international AI talent; deliver upgrades to digital and data infrastructure; ensure a business climate conducive to starting and growing an AI business; and contribute to prosperity of society by spreading AI benefits across the country [1]. The AI Sector Deal outlines the commitment from government, industry and academia as the three partners with a sectoral support package of around £1bn that complements an additional £1.7bn under the Industrial Strategy Challenge Fund [1].

This cross-sectoral partnership is also built in the governance structure of the AI Sector Deal [1]. Oversight of the implementation of the Deal and maximization of its potential will be led by the new Government Office for AI. The office will support the AI Council that will bring together leaders from industry and academia to provide strategic leadership and drive the implementation of the Deal. Societal benefits of AI will be ensured through the creation of a Centre for Data Ethics and Innovation to advise on the ethical use of data and AI.

For the future industry of the UK, the AI sector has the potential to provide over 80 000 new jobs [1, p. 36], adding £200bn or 10% of UK GDP by 2030 [1, p. 24]. AI also holds significant promise for a public sector that is undergoing transformation with robotics and automation changing the provision of public services [2]. The challenge for AI adoption in the public sector is to better use citizen data for improvement of public services. The direct value of citizen data held in the public sector has been estimated at £1.8bn, with wider social and economic benefits totalling £6.8bn [3]. These data can be used to more specifically target who needs public services and ‘tailor those services more accurately’ [2].

Governments have historically taken on the role of an entrepreneurial state playing a significant role in innovation [4]. A well-known example of such initiatives is the Defence Advanced Research Projects Agency (DARPA) of the USA, which has developed the technology behind the Internet and personal computer [5]. The UK has also seen multiple examples of how this can be implemented in practice, with initiatives such as Innovate UK, the Small Business Research Initiative (SRBI), and The Catapult Programme, among others.

The ambition of the AI Sector Deal is amplified by external events like Brexit, but ultimately depends on successful collaboration among the three partners. Such collaboration has become a fundamental activity in most, if not all, entrepreneurial State initiatives [6]. Public organization adoption of AI and data science presents numerous known challenges ranging from employee path dependency on embedded processes and norms, information silos, a lack of resources, collaborative culture and technical capacities [7,8]. Successful delivery of the AI Sector Deal relies on collaboration across the three partners. It is, therefore, important to understand the challenges and success factors for such cross-sectoral collaboration, learning from collaborative experiences in other sectors. The present study synthesizes existing knowledge on how to manage cross-sector collaborations and proposes a series of recommendations on how they should be considered to integrate AI and data science initiatives into public service delivery.

2. Mapping inter-organizational collaboration in artificial intelligence

Delivery of public services is often implemented via different types of organizational forms bringing together public, private and non-profit actors [9]. These collaborations may imply highly formalized ventures, such as public–private partnerships, or more informal arrangements, such as policy networks. Their main rationale is to merge the strengths of each involved party to increase the effectiveness and the value for money of a particular public service. Cross-sector collaboration has gained importance worldwide, particularly across European countries [10]. In the UK, for instance, more than 725 public–private partnerships worth over £54.2bn have been developed to create hospitals, schools, prisons, bridges, roads and military equipment [11]. Despite their popularity, collaborations entail numerous management complexities [12,13] and, as a result, a high percentage of them frequently do not achieve satisfactory outcomes [14], while some are highly successful [15].

There are existing tri-lateral collaborative arrangements in the area of AI and data science that provide the knowledge base. For example, the University of Essex arranged a joint appointment with Essex County Council of a professorship in the area of public policy and data science as Chief Scientific Adviser to the Council that is based in a specially designated institutional vehicle—the Institute for Analytics and Data Science (IADS) within the School of Computer Science and Electronic Engineering. IADS is a centre of excellence at the University of Essex that connects scholars, businesses, institutions and government for AI-related work. The aim of the relationship is to leverage resources and data of the public sector with AI expertise of the University and businesses to deliver services for the benefit of the community in Essex. Table 1 provides an overview and summary of several additional examples of similar collaborations from around the world.

Table 1.

Cross-sectoral AI and data science centres of excellence.

centre	participants	participant type	aims	core areas of application
Institute for Analytics and Data Science (IADS)	University of Essex, Essex County Council, Suffolk County Council, British Telecom, EPUT NHS, UNESCO	university; local government; NGO; organization; private utilities	to create new products and services for businesses, individuals and society; to facilitate knowledge transfer around AI between academia and private, public and third sectors	international development, public policy, healthcare, social care, mental health, insurance, finance, telecoms, transport, media, policing and crime prevention
location: UK
Singapore Data Science Consortium (SDSC)	National University of Singapore, Nanyang Technological University, Singapore Management University, Agency for Science, Technology and Research, National Research Foundation Prime Minister's Office, Defence Science & Technology Agency, Singapore Tourism Board, ST Electronics, GIC, Micron, Fuji Xerox, Surbana Jurong, Certis Cisco, ASM Assembly Systems, Television Content Analytics TVCONAL	universities; national research agencies; private tech companies; local government	to facilitate collaboration between institutes of higher learning, research, industry and government in data science R&D	healthcare, customers and retail, manufacturing, transport
location: Singapore
AI Singapore	National University of Singapore, Singapore University of Technology and Design, Nanyang Technological University, Agency for Science, Technology and Research, Singapore Management University	universities; national research agencies	to catalyse, synergize and boost Singapore's AI capabilities, to use AI to address major challenges that affect society, and to invest in deep capabilities to catch the next wave of innovation	healthcare, urban mobility, cybersecurity, computing platform, privacy preserving technologies, sensing and measurements
location: Singapore
Beijing Institute of Big Data Research (BIBDR)	Peking University, Beijing University of Technology, Zhongguancun Science Park, Haidian District government under supervision of municipal government of Beijing	universities; district government	to combine education, research, entrepreneurship and government service to create world-class programme for developing data science in China and a platform for nurturing new enterprises in big data	healthcare, traffic, finance
location: China
RMIT Data Analytics Lab	RMIT University Melbourne, NICTA (NSW government, Queensland government), Australian Research Council	universities; regional government; national research council	to become a hub for advanced data analytics projects to help Australian business compete on a global scale	geospatial information search, biomedical informatics for health decision making, integrated design infrastructure for Australian cities
location: Australia
The GovLab- NYU	NYU Tandon School of Engineering, White House Office of Science and Technology, Laura and John Arnold Foundation, MacArthur Foundation, The Australian National Government, UK's National Health Service, UNICEF, Omidyar Network	universities; national research institute; private foundation; foreign government partners; NGOs	to strengthen the ability of institutions and people to work more openly, collaboratively, effectively and legitimately to make better decisions and solve public problems with big data and open data	criminal justice, healthcare, government innovation, public decision making
location: USA
California Policy Lab	UCLA, UC Berkeley, Californian governments local, county and state levels	universities; state departments; county government; local government	to create data-driven, scientific evidence and insights to help government at all levels in the state solve urgent problems; to help bridge the gap between policy makers in the research community	homelessness, poverty, crime, education inequality
location: USA
Center for Data Science and Public Policy	University of Chicago Harris School of Public Policy, Computation Institute, Municipality of Rotterdam, Charlotte-Mecklenburg Police Department, Metropolitan Nashville Police Department, San Francisco Police Department, Los Angeles Sherriff's Department, Chicago Department of Public Health, Chicago Department of Innovation and Technology, Environmental Protection Agency	universities; local government; county government; research institute; public research; national government	to educate current and future policy makers, doing data science projects with government, non-profit, academic and foundation partners, and developing methods and open source tools that support and extend use of data science for public policy and social impact	welfare, city infrastructure, citizen engagement, highway patrol, urban planning
location: USA
Dalle Molle Institute for Artificial Intelligence (IDSIA)	Swiss Confederation Commission for Technology and Innovation, University of Lugano, University of Applied Sciences and Arts of Southern Switzerland, Imprecise Probability Group (IPG), Swiss National Science Foundation, Federal Department of Defence	universities; national research institute; national government; research network	to offer solutions to a range of complex problems through theoretical findings and novel algorithms, machine learning, deep neural networks, and imprecise probabilities by promoting strong cooperation with partners	military decision making, metrology and climatology, environmental risk analysis, bioinformatics
location: Switzerland
German Research Centre for Artificial Intelligence (DFKI)	University of Bremen, Deutsche Forschungsgemeinschaft, Deutschland Land der Ideen, Berlin Big Data Center	universities; national government; research institute	to study design, realization and analysis of information processing models that enable robotic agents and humans to master complex human-scale manipulation tasks that are mundane and routine	emergency response and crisis management, outreach, multimedia opinion mining
location: Germany
Insight Centre for Data Analytics	Dublin City University, NUI Galway, University College Cork, University College Dublin, Cisco, Intel Corporation, Tyndall National Institute, HP, Central Statistics Office, Open Data Institute, Dublin City Council, Galway City Council, Department of Public Expenditure and Reform	universities; councils; national government; research institute; private sector	to use information to make decisions based on it for transformation by taking the guesswork out of decision making in society	personalized public services, chronic disease management and rehabilitation, smart enterprise, open government, urban life quality
location: Ireland
EBTIC	Khalifa University-Abu Dhabi campus, ICT fund-Telecommunications Regulatory Authority, Etisalat, BT	universities; private utilities; national government	to collaborate with industry, universities and government organizations to be a driving force for innovation for the Middle East region	smart infrastructure, smart network design, smart society, smart enterprise
location: UAE

Table 1 illustrates that, across continents, governments are engaging with universities and a variety of sectors through policy laboratory platforms in order to combine different capabilities and problem-solving capabilities. These cross-sectoral laboratories help synergize knowledge so that government can work towards AI- and data science-based solutions in areas ranging from prevision medicine to smart cities. For instance, The GovLab at NYU frequently develops applied research frameworks that help government approach problems in a more data-informed, innovative manner. The Lab's People Led Innovation Methodology is used by city officials to approach major public problems through a series of four phases that unleash the expertise of others to create solutions. These AI and data science laboratories often provide a network for training and skill development for public servants and recommend communication and technical advice for ‘smarter’ management.

The benefits of adopting AI drive the incentives for collaborative arrangements. These benefits relate to prediction and anticipation of demand for services, automation of demand-side response, identification of high-risk groups and development of targeted interventions; production of goods with higher productivity, lower cost and better efficiency; promote products and services at the right price, with the right message, and to the right targets; and provide enriched, tailored and convenient customer experience [16].

Social benefits of such collaborations focus around improving public service delivery and relieving administrative burdens. For example, Essex County Council predict the risk of 14-year-olds becoming NEET (not in education, employment or training) by age 18 and work with schools to develop early-stage interventions with additional support to encourage those at high risk of becoming NEET to remain in employment or education [3, p. 37]. The UK Cabinet Office Behavioural Insights Team showed how to analyse initial referral and assessment notes in social care to predict closed case escalation (how many cases would come back into the social care system) [17]. Through overcoming resource constraints, paperwork and backlogs in a more cost-efficient, effective and time-saving manner, government gains the opportunity to exist as an empathetic service provider [18]. To illustrate, DARPA's ‘Education Dominance’ programme uses AI to reduce the time required for Navy recruits to become technical experts (from years to months) through the creation of a digital tutor that applies machine learning to model novice–expert interaction [19]. This resulted in the programme recruits outperforming experts with 7–10 years of experience and ensured the recruits' likelihood of securing high-tech jobs with high incomes.

Cross-sectoral collaborative efforts around AI are often institutionally organized around offices for data analytics (ODAs). Table 2 provides a broad overview and summary of major international efforts in this area.

Table 2.

Cross-sectoral data analytic centres.

office of data analytics	participants	participants	aims	core areas of application
City of Boston Analytics Team	EMS Boston, Office of New Urban Mechanics, Boston Fire Department, Inspectional Services Department, 311 Call Center, Data for Democracy	local government; non-profit	to act as a central data organization interested in using data and maps to create a better understanding of Boston and to use data to improve city public policies	geospatial, city services, permitting, environment, transportation, emergency response, citizen engagement
location: USA
DataLA	Ash Center for Democratic Governance and Innovation at Harvard Kennedy, USC Spatial Science Institute, UChicago CDSPP, UCLA, City Parking, County Department of Public Health, County of Los Angeles Bureau of Land Management, LA Sanitation Department	universities; regional government; local government	to work with academics, city departments, the community, sister cities and private partners to develop insights and digital tools that make the city more liveable and equitable by sharing data	publishing and maintaining open data, geographical data and communication, Open Budget LA, street cleaning efficiency
location: USA
MODA NYC Analytics	Department of Information Technology and Telecommunications and Technology, NYU Center for Urban Science and Progress, Columbia University Institute for Data Sciences and Engineering, Department of Citywide Administrative Services	universities; local government; national government	to partner with agencies to create, test and improve analytic models that deliver measurable value to city services and to serve as NYC's civic intelligence centre	crime, public safety, quality of life issues, citywide data sharing platforms, training and skill development for city employees
location: USA
DataSF	Open Data Services Team from Department of Technology, Department of Health, San Francisco Department of Environment, San Francisco Art Commission, Code for America	local government; state government; non-profit	to empower the use of data across city departments so that evidence-based policy making and operational improvements can be made	cross-departmental open data sharing, data governance, data quality standard setting, mother and child nutrition, training and skill development for city and county staff, city greening
location: USA
SmartDubai	Mohammed Bin Rashid School of Government, Dubai Health Authority, Dubai Police, Roads and Transport Authority, Department of Economic Development	university; local government; national government	to collaborate with private sector and government partners to empower, deliver and promote efficient, safe and impactful city experiences for residents and visitors	data leadership skill development and training, mapping city infrastructure, e-services, environment, mobility
location: UAE
Office of Open Data and Digital Transformation-Philadelphia	University of the Arts Design for Social Impact Program, Division of Housing and Community Development, Department of Planning and Development, Penn Medicine's Center for Health Care Innovation	university; local government; state government	to create digital services that support the success and well-being of all Philadelphians to empower them through dignified, accessible and efficient services	public open data, human catered service design, data sharing platforms, citizen engagement, housing accessibility, historic site vulnerability, public health
location: USA
Greater Manchester Connect	University Hospital Morecambe and Cumbria Information, Transport for Greater Manchester, Manchester City Council, Health Innovation Manchester	regional government; local government	to put Greater Manchester at the forefront of data sharing and analysis to help improve public services by establishing a data sharing authority to break down the barriers which stop public services from sharing information	health and social care and wider reforms of public services, information governance platforms, data sharing engines, employment and skills, housing, transport
location: UK
ODA for the West Midlands	GDS, Department for Communities and Local Government	national government; local government	to bring together investment in research, data and intelligence to support the delivery of the region's Strategic Economic Plan, and provide an evidence base for future changes in public services	open data
location: UK
Government of South Australia ODA	Australia Bureau of Statistics, Department of the Premier and Cabinet, Australia Renewable Energy Agency, Clean Energy Council, Bureau of Meteorology	national government	to provide high-quality data analysis and to support South Australia government agencies and better the State of Australia	child protection, gender equality, domestic violence, energy
location: Australia
SmartDublin	Dublin City Council, South Dublin City Council, Fingal County Council, Comhairle County Council, Intel, IBM, Maynooth University, Lero, Insight	local government; regional government; private tech companies; universities	to create a mix of data-driven, networked infrastructure, fostering sustainable economic growth and entrepreneurship, and citizen centric initiatives	energy monitoring, public transportation passenger information, civic engagement and citizen empowerment, dashboards, trash, traffic
location: Ireland

ODAs are commonly used as a collaborative organizational form for sharing data to improve city services. What differentiates ODAs from policy laboratories is that policy laboratories operate as a networked platform often rooted in universities, whereas ODAs are physical offices often associated with the mayor or city manager's office. One example is the Mayor's Office of Data Analytics in New York City (MODA NYC) which serves as a unifying space for aggregating data from across city agencies to more effectively address crime, public safety and quality-of-life issues. One contribution from MODA NYC is the creation of a citywide data-sharing platform, Databridge, which ‘combines automated data feeds from 50 plus source systems across 20 agencies and external organizations to warehouse and merge geographical information to enable cross-agency analysis’ [20]. While ODAs commonly work across agencies within government, table 2 portrays how other organizations often contribute to knowledge sharing and analysis in these settings. In order to develop standards and protocols for data sharing, MODA NYC actively partners with the expertise of NYU Center for Urban Science and Progress as well as Columbia University Institute for Data Science and Engineering. Thus, ODAs also take on a ‘data liaison’ role as a ‘designated point of contact for outside partners contributing to or using city data’ [20].

Such cross-sectoral collaboration is as promising as it is challenging. While the benefits of interacting across sectors to implement AI strategies are many, working across sectors has been proven to be very complex [8]. The next section discusses challenges and success factors of cross-sectoral collaboration that should be considered for successful delivery of the AI Sector Deal.

3. Challenges and opportunities of cross-sectoral collaboration around artificial intelligence

In order to analyse the managerial practices influencing collaborative arrangements across different sectors, we used a systematic review of the existing literature. The search strategy used to find eligible studies was carried out with an electronic search in the Google Scholar database. We chose our publication criteria to be only those public administration and management articles found in, arguably, the top three journals in the field: Journal of Public Administration and Theory, Public Administration Review and Public Management Review. The following search terms were used: ‘collaboration performance’, ‘collaboration success’, ‘network performance’, ‘network success’, ‘joint venture performance’ and ‘joint venture success’. These searches led to 7885 results initially; once the journal filter was added, 156 results were acquired. Next, after searching the abstract and title for the relevant terms, a total of 84 articles were included in the review. We also used a ‘study design’ filter and kept only research with empirical evidence (e.g. articles that use research design such as case studies, surveys, questionnaires) on factors for success and performance in collaboration, networks and joint ventures. For each article, we summarized the authors, publication year, title, journal, success factors identified, effectiveness determinants identified, and managerial strategies that led to success. By examining only articles with the keywords ‘success’ and ‘performance’, we were able to extract determinants that led to success as identified in the empirical studies. In order to analyse the selected studies, we engaged in an inductive analytic process [21] to derive the main factors influencing success in collaborative ventures.

(a). Challenges for successful collaboration

While the challenges of collaboration across private sector organizations have been widely researched [22], much less attention has been paid to the difficulties of working across public, private and non-profit sectors. Scholars such as Stoker et al. [23] have focused on the development of social capital as a means to address cross-sectoral collaborative challenges that are rooted in the way actors perceive each other's abilities to relate to one another. Additionally, in the environmental policy field, Innes et al. [24,25] have contributed insights about the benefits of network structures in overcoming traditional bureaucratic-based institutional constraints through their self-organizing and adaptive nature.

More recently, Andrews et al. [26] reviewed empirical works on the differences among public and private organizations and proposed a series of arguments on how these differences challenge inter-organizational collaborations, based on mixing environments, structures, goals and values. The first issue that can hinder collaboration success is the different environments surrounding public and private organizations. While public organizations are accountable to their service users and, also, to the public at large, private organizations respond to their shareholders [27]. This can lead to clashes when aligning the interests of the different partners engaged in the collaboration [15,26]. In a recent report on AI by the House of Lords [28], new questions of accountability were raised. Public sector procurement of AI-based technologies presents challenges regarding the ‘legal liability where a decision taken by an algorithm has an adverse impact on someone's life’ or ‘the potential criminal misuse of artificial intelligence and data’ [28, p. 95]. There is a duality surrounding the positive impacts that data-based decision-making tools and machine learning can have on public policy making and implementation. If things go wrong, pinpointing responsibility becomes a web of closely inter-linked realities: ‘Is it the person who provided the data? The person who built the AI? The person who validated it? Operates it?’ [28, p. 309]. Additionally, in the case that the business which creates the technology is responsible, it is not unlikely these companies exist overseas, in places like China or Singapore, potentially turning jurisdictional action into a sand-trap of international law.

Another central aspect of the challenges associated with mixing environments in cross-sectoral collaborations is the divergent approaches to managing risk in the public and private sectors. Klijn & Teisman [29] argue that the political risks of government are not easily reconciled with the market risks of business organizations. As future inter-organizational collaborations related to AI take place, there is always the inherent risk that the data used have been gamed or sabotaged to serve the opportunism of a self-interested actor. For example, this ‘needle in the haystack’ situation can occur in training or operation phases like the intentional use of misleading data fed into systems or ‘destroying, altering, and injecting large quantities of misleading data’ [28].

Competing institutional logics have been considered a fundamental challenge when developing collaborative ventures [30]. Public organizations operate in what has been described as a state logic, while private organizations operate with a mix of market-based and corporate logic. These authors expose that business partners in collaborations ‘conflate their role as shareholders—thus invoking the market logic—and their experience as businessmen, as they are accustomed to operating under the Corporate logic within their companies’ [26, p. 347]. In practice, this influences the agreement of which goals need to be pursued by the collaboration. In particular, managers of collaborative ventures may find it difficult to deliver public value for money, and also to maximize profits to satisfy shareholders of the private partners [31]. Another important aspect affecting collaboration success is the mix of different organizational structures. In this sense, public sector organizations are classically identified as rule-orientated because of the need to meet demanding statutory requirements for due process. By contrast, Rainey [32] describes how private firms are not subject to the same kind of political accountability pressures and so are thought to be less hampered by bureaucratic oversight.

Opportunism in strategic collaborations has been linked with divergence among the organizational values of all involved partners [33]. This is of special importance across cross-sector collaborations, as each sector has been related with a subset of different values—i.e. public employees are more motivated to serve the public, while their private counterparts seek to further their organization's interests [34]. In a nutshell, then, the major challenge caused by the mixing of values in collaborative ventures involving public, private and non-profit sectors is to help the members of each organization to switch their mentalities from the ‘us and them’ to ‘we’ [35]. For instance, while cross-sector collaboration forms for AI and data science can be commonly based on procurement or contracting out, university–public sector forms present an opportunity to unite organizational differences through mission-oriented projects. The hope is that projects centred on addressing societal problems will align similar values between organizations due to the ‘social good’ nature of these projects.

Elaborating on this sentiment, Healy [36] argues that the fragmentation of values in collaborative governance can be united when ‘substance and process’ are recognized as ‘co-constituted, not separate spheres’ [36, p. 112]. The engagement in the governance process ‘shapes participants' sense of themselves; and generates ways of thinking and acting that may be carried forward’ with the emergence of a social order [36, p. 112]. Creating opportunities to overcome the value challenge is a key task for public managers. To illustrate, The White House Police Data Initiative uses inter-organizational collaboration (e.g. engagement with academics, technologists, police departments) to experiment with machine learning techniques that review audio and video footage from body cameras [37]. Here, the data presented from body cameras serves as a vehicle for knowledge sharing among different actors like Stanford University academics and the City of Oakland police department over the benefits and problems of this tactic. From this knowledge sharing process and exchange of perspectives, social and relational substance begins to bridge individual and institutional differences towards collective values and action.

Additional challenges related to cross-sectoral collaboration around AI relate to skills and data. There is a significant skills gap in AI between the public sector, on the one hand, and businesses and universities, on the other. The AI Sector Deal is providing for significant investment in skills and people for the wider UK economy. However, there is less consideration of the AI skills gap in government (and the wider public sector). Moreover, Chen et al. [38] associate public organizations as lagging in individuals who possess the ‘prerequisite knowledge and skills to be effective participants’ in cross-boundary data-based initiatives, and thus require technical assistance and training. In particular, these issues relate to a lack of the technical jargon needed to personalize ‘disparate public data’ from different organizations so that it tells a story; user ability to operate new systems; and incentivizing information sharing at the individual level. At the same time, developing the digital skills needed for public sector use of artificial intelligence is not a quick process, and more funding is needed for PhD students in machine learning to overcome this general shortfall [28]. Extending the system of secondments across the three partners can be an immediate solution that could also kick start the knowledge transfer. Classically, Weber et al. [39, p. 335] find that the ‘transfer, receipt, and integration of knowledge across participants’ is a constant challenge for any public problem being addressed in inter-organizational settings. Hence, managers must strategically encourage employees to share new information and skill development with co-workers ‘to enhance the collective improvement of knowledge’ [40, p. 699].

Open data initiatives have been largely successful in unlocking commercial value from publicly held data. However, some of the most valuable data for AI innovation cannot be openly shared due to commercial sensitivity, security or personal information. Successful development of the AI sector relies on developing deeper data sharing relationships across the three partners, with the barriers ranging from trust and cultural concerns to practical and legal constraints. Poorly implemented data sharing programmes risk derailing innovative AI cross-sectoral collaborations as witnessed from the case of DeepMind and the Royal Free London NHS Foundation Trust [28,41]. The AI Sector Deal aims to address this issue through the establishment of Data Trusts providing clear frameworks for fair, equitable and secure data sharing [1, p. 30]. Without this type of central information system, the technical capacity to share data across inter-organizational forms is hindered by the fragmentation of data standards [38]. Scholars also suggest that in addition to data sharing standards, forming data collection standards and data quality assurances before it is shared among organizations further augments the technical capacity for joint action.

The establishment of a Centre for Data Ethics and Innovation will also ensure safe and ethical use of AI, while the General Data Protection Regulation (GDPR) and the UK Data Protection Bill provide legal certainty over the sharing and use of data, and fair and transparent application of AI [1]. Beyond concerns of ethical use of AI and fairness in the sharing and application of AI, unintended consequences also relate to the ‘consequential decisions about people’ often made by humans that will be replaced by AI and safety as more ‘AI [is used] to control physical world equipment’ [42, p. 30]. Specifically, the report raises concerns about how to ensure justice, fairness and accountability in this decision making and how machine learning systems will react to the ‘complexities of the human environment’. Especially in relation to the criminal justice context, institutions such as the Centre for Data Ethics and Innovation must constantly push for the incorporation of data that is as complete and unbiased as possible. Otherwise, we risk ‘exacerbat[ing] problems of bias’ into these new technological interfaces and ‘hardwire discrimination’; however, data analytics can also be used to ‘predict and detect bias and prevent discrimination’ [43].

Andrews et al. [26] conclude by highlighting the many potential benefits of cross-sector collaborations but warn that the management complexities that they entail hinder, in most cases, the possible benefits of bringing the strength of different sectors and combining them in a particular project. Thus, understanding the success factors related to collaborations across sectors, together with the different managerial approaches to mitigate the above-mentioned challenges, becomes of great importance if initiatives related to, for example, AI projects, need to be implemented through public, private and non-profit collaborations.

(b). Success factors of collaboration

Research on the performance of collaborations is vast [44–46]. One of the main concerns within the literature on collaboration performance has to do with the managerial strategies that can increase the performance of these complex organizational arrangements [47]. Among the reviewed studies, seven main managerial strategies can be distinguished: facilitative leadership; shared objectives; knowledge gathering and sharing; communication; socializing; expertise; and sense-making.

• (1) Facilitative leadership: Opposite to the classic idea of a hierarchical leader who imposes his or her views upon followers by relying on his or her power position within the organization, facilitative leadership ‘endorses respect and positive relationships among team members, constructive conflict resolution, and candid expression of thoughts and attitudes’ [48]. Ansell & Gash's [49] meta-analysis of the literature on the management of collaborative governance concludes that leaders of collaborations should promote broad, active participation; ensure broad influence and control; facilitate productive group dynamics; and extend the scope of the process. Therefore, it is argued facilitative leadership is imperative to collaboration, especially since incentives to participate can be low and resources may often be asymmetrically distributed. Another implication is that the authors implicitly derive that collaboration performance is determined by achieving a cycle of communication, trust, commitment, understanding and outcomes. The success of collaboration is implicitly contributed to a combination of face-to-face dialogue (although this alone is not sufficient), the ability to establish trust in the various phases from negotiation to implementation, the level of commitment from stakeholders (which requires cooperation and responsibility to results of consensus and ‘ownership’ of the decision making), as well as a shared understanding of what can be achieved through working together viewed as common ground or common purpose. It should be noted, however, that facilitative leadership style is not the only leadership approach to manage collaborative ventures. In a recent analysis of cross-sectoral collaborations to deliver water services in Norway, Hovik et al. [50] found that the leadership styles can be contingent to the partnership's characteristics, and the characteristics of the manager determine their ability to understand how to leverage different skills needed. These authors identified four main roles of collaborative leaders: the convener who ensures information flow, role clarification and compliance; the catalyst who creates motivation, raising awareness and ensuring ownership; the mediator is the broker and facilitates discussion; and the bridge builder links aims at different levels and ensures political anchorage.

• (2) Shared objectives: The definition of the organization's objectives has been positively correlated with the organizational performance of the public sector [51]. However, since alliances involve both joint value creation and value appropriation, these mixed motives may create tension between shared and private objectives [52]. The simultaneous pursuit of different objectives leads managers to continue working in the ways they were used to, because they do not know what objectives to focus on [53]. This lends some support to Jensen's [54] assertion that asking managers to pursue multiple objectives creates problems—not ‘confusion and lack of purpose’ as he suggests, but rather a ‘status quo bias’ [53]. Even if all the parties in a collaboration are highly aligned with the main objective of the alliance, there may be differences between the objectives of each organization. The importance that objectives have for collaborations is explained because they ‘act as a guide for decision making and a reference standard for evaluating success’ [55].

• (3) Knowledge gathering and sharing: To overcome process and dynamics issues of collaborative governance, Chen & Lee [38] suggest management activities should focus on institutional capacity building for joint action, like the creation of common standards for the collection and processing of data. On a technical level, their in-depth case study finds that the federally mandated metropolitan planning organizations are challenged by the management of the collaborative data networks necessary to create data sharing across jurisdictions, which is required for more integrated metropolitan transportation planning. The main implication is the way in which knowledge is represented across the divisions of functional departments can nonetheless enable or hinder the improvement of cross-boundary data sharing. Therefore, the formulation of common standards for data collection and sharing is best developed by activating key network members, so that groups are aligned based on their functional responsibilities across the network [38]. For example, front-line workers are knowledge banks who can support and recommend the design of procedural standardization. The authors found that the GIS members maintain regular contact and communication in regional transportation planning activities and can contribute specialist insights of data and technology operation. To this end, NESTA [56] recommends that in collaborating with the Greater London Authority and data science specialists to develop an algorithm that predicts which of the City's thousands of properties are unlicensed ‘House(s) in Multiple Occupation’, the first step was to speak to building inspectors about the features on a typical HMO, due to their front-line knowledge competencies. Furthermore, it is suggested that improved data sharing procedures derived from using that information could then identify the relevant datasets connected with these criteria. Accurately building the institutional and technical capacity to guide the collaboration requires incorporating quality knowledge at the beginning of the process, so that fragmentation will not ‘create issues later on for data integration’ [38].

• (4) Communication: In a recent study, Ansell & Gash [57] have described the different effects that a communication strategy can have on the management of a collaboration. First, they describe the attractor effect, which occurs when it appears that the collaboration is producing tangible outcomes, so stakeholders are more willing to invest time, energy and resources. This happens by showing the value of joint action through quick wins. Positive learning feedback is another determinant of success in that creating ‘politically neutral’ spaces for joint learning reduces cultural barriers and increases networking [58]. Learning feedback thus occurs when the knowledge gained from learning how to work together is continuously built upon in subsequent interactions. Next, collaborative platforms are successful when they can exercise architectural leverage, which is achieved through developing shared assets, designs and standards that can be reconfigured, resulting in multiplier effects built on these pre-existing efforts. Similar themes of performance determinants from past literature are also cited, such as a ‘champion’ to mobilize support and activity coordination.

• (5) Socializing: When managers make the impact of the efforts of the collaboration transparent and enticing for key players to work together, the collaboration will be positively affected [59,60]. This seems parallel to the beneficiary contact findings that when employees see how their results impact a person, they increase performance. Transparent results and indicators can facilitate more ideas and reforms throughout all levels of the collaboration where it may be more difficult to implement a top-down idea in decentralized settings. The US healthcare sector has been experimenting with this more holistic approach to understanding the ‘whole’ system surrounding problems in their reimbursement payment schemes through the use of big data collaboration [61]. By combining insights of health data from clinicians, researchers and patients, there is a shift away from ‘isolated and potentially uncoordinated instances of treatment- or fee for service-[towards] paying on the basis of better health outcomes’ [61, p. 22]. In contrast to a former fragmented analysis, an interconnected ‘learning system’ is quicker to transfer knowledge from different levels back to providers. As Page [60] states, ‘inclusive processes used to design and deploy the results and indicators help ally partners' mutual suspicions and turf differences at the beginning of the reform process’ (p. 333). This democratizes the power in the setting in a way, because, once it is known where everyone stands, the next phase of the collaboration can use that information to build upon their practices or efforts and not just keep the results isolated to the reform architects. This is further explained ‘because the vision, mission, goals, and daily activities of collaboration transcend particular individuals and organizations, the results and indicators were necessary to foster organizational commitments to common values and practices’ [60, p. 333]. Thomson & Perry [62] complement these ideas by defending the importance of shared responsibility. Arguably, arriving at the general consensus needed to manage the collaboration requires an equilibrium where conflict can still occur but within a larger framework of a jointly determined agreement about the rules.

• (6) Expertise: Overall, the manager's mind-set determines the choices about when and how to use analytical tools and strategies needed for the transfer, receipt and integration of knowledge across the network [39]. More generally, hiring tech-savvy network managers and shepherding the efforts of field experts within the network can both induce trust based on their competencies, as well as improve the quality of service [63]. From an operational standpoint, Chen [38, p. 15] exemplifies that ‘the appropriate use of relevant technology can significantly improve performance in data quality, data integration, data analysis, and visualisation’. In an analysis of best practices surrounding data-based collaborations in the public sector, NYU GovLab [64] suggests that managers should tackle a lack of institutional readiness by ‘utilising the host of data legacy managers already working in government’ like GIS teams for their wealth of information and tactical expertise. From piloting the London Office of Data Analytics, Eddie Copeland [56] also highlights in a lecture that managing downwards in public organizations can liberate the already talented data analysts, who are likely ‘stuck reporting on monthly dashboards and key performance indicators’, instead of being used in other administrative capacities. Strategies for integrating the correct knowledge for data analysis in public sector collaboration can further be enhanced through engagement with experts in the field. In particular, during a project with the Municipality of Rotterdam's Rijkswaterstaat traffic patrol department, the University of Chicago Centre for Data Science and Public Policy [65] attributes their first-hand observation of how the relationship works between inspectors patrolling the highways, and traffic control centre's reaction, as a means for designing a more successful data-driven approach to the deployment locations for patrolmen. As legitimacy is a key factor in network success, collaborative managers would benefit from ensuring that expert powers from within are properly used to infuse credibility among the transfer, receipt and integration of data-driven knowledge [60].

• (7) Sense-making: Heen [66] explores the performance outcome of satisfactory delivery of primary care medical services and the impact that different managerial roles can have on this success. In her case studies, the municipalities are dependent on cooperation from the regular GPs to solve issues like securing patients, while the GPs' interest is that the network enables them to influence municipal decision in their sector. Thus, the author finds the relationship to be asymmetric: the municipality has more need for cooperation with the GPs than vice versa. The findings from the case studies exemplify that the context of unbalanced reciprocity denotes situations for more game-like activity of indirect management attempting to create strategies for trust building and persuasion. A collaboration manager must, then, make sense of the situational need. For example, fragmentation requires this role to stimulate the formal network structure through enlivening actors to engage themselves. Trust in the diplomacy role is often grounded in authority and tied to the perception that an actor has influence. Next, it is argued that, as conflict is inevitable, an adversarial role of management can arise due to the mandatory contractual nature of the network. Because the adversary would openly challenge and confront the opposing parties as someone who is less interested in brokering, and more concerned with maintaining a representative role of the regular administration, distrust is directed towards this figure as a symbol of the municipality, rather than in a personal way. Finally, the partner network manager role indicates that a network has institutionalized, and actors have become integrated, but still requires nurturing. In a similar vein, studies have provided insight into managerial interventions that can be used at each stage in the process of collaboration, highlighting the importance of sense making when choosing which managerial role is needed at each stage of the collaboration [39,67].

Table 3 summarizes the literature on the management of collaborations involving the public sector. The aforementioned seven management strategies for inter-organizational collaboration were elaborated on from the findings in this table.

Table 3.

Management strategies for inter-organizational collaboration.

author(s)/year	management action	results
Ansell & Gash (2007) [49], Geddes (2012) [68], Hovik et al. (2015) [50], Klaster et al. (2017) [69], Waugh & Streib (2006) [14]	Designate facilitative leadership at various levels and stages of collaboration (e.g. boundary spanners, ‘champions,’ during negotiation, etc.)	Conflict resolution, consensus building, action, inclusive agenda shaping, broad participation, productive group dynamics, empowerment, unity of purpose and an extended scope
Agranoff & McGuire (1999) [70], Ansell & Gash (2017) [49], Chen & Lee (2018) [38]	Promotion of joint action building through creation of shared standards and goals	Develop institutional capacity through less institutional and technical inhibitions
Ansell & Gash (2017) [49], Bryson et al. (2006) [71], Saz-Carranza & Ospina (2012) [58]	Create learning spaces, a communication strategy and a compelling vision	Leads to reduced cultural barriers, the development of a sense of commonality among stakeholders and helps to overcome tensions
Cuganesan et al. (2017) [59], Page (2003) [60], Thomson & Perry (2006) [62]	Induce sharing and stewardship through providing information about skills, resources, policies and examples; make impact of collaborative efforts transparent to create symbols of progress	Employees will change their mind-set in desired way through self-efficacy, certainty and legitimization; this stimulates cohesion, innovation and ability to reframe meanings to achieve shared and independent goals
Giest (2015) [63], O'Leary et al. (2012) [72], Chen & Lee (2018) [38], Weber & Khademian (2008) [39], Agranoff & McGuire (1999) [70]	Use experts to facilitate the demands of highly specialized networks	Expert knowledge helps frame tasks and alternatives ways of conceptualizing problems
Heen (2009) [66], Vangen & Winchester (2014) [67], Weber & Khademian (2008) [39]	Understand the situational need of management styles	Adopting practices can help positively control the impact activities have on the diverse culture and power balances

In sum, the main implications for these strategies include increased conflict resolution, inclusive agenda shaping, institutional capacity, unity of purpose and power balance.

4. Conclusion

AI holds significant potential to contribute to an ambition to make the UK the world's most innovative economy. In order to fulfil the potential of AI for the UK economy and society, the government is expected to take on an entrepreneurial role in innovation policy through cross-sectoral collaborations with businesses and universities. Examples of such collaborations are already appearing both in the UK and internationally. For example, the Institute for Analytics and Data Science at the University of Essex focuses on delivering AI transformation across the public services in Essex.

AI collaboration may be new, but we have accumulated a lot of experience with similar type ventures in various economic sectors. The potential benefits of bringing public, private and non-profit actors to collaborate for public service delivery are well known [73]. Nonetheless, these benefits do not come alone. Existing evidence from these types of collaborative ventures suggests that we should be aware of vast managerial complexities and their negative effects on the effectiveness and the value for money of cross-sector collaborations. International evidence suggests that these difficulties do not affect the increasing use of collaborative ventures to deliver public policies across the globe [15]. However, they do contribute to the fact that, in several cases, these organizational forms do not achieve the desired results [74].

In order to achieve the highest potential of the AI cross-sectoral collaboration as suggested in the AI Sector Deal, the success factors of similar enterprises around the world must be considered. In a nutshell, we find that facilitative leadership is imperative to collaboration success. In addition, alignment of goals and objectives between all the involved parties has also been identified as a key factor for collaboration success. This is of particular importance when organizations from different sectors collaborate, as managers can use how knowledge is represented through the creation of shared standards to promote joint action for institutional and technical capacity building. A well-defined communication strategy will certainly help to align the interests and expectations of all the members of the collaboration, especially when it comes to discussing the opportunities that data science and AI can bring to a particular project. Socialization has also been identified as key factor of cross-sectoral collaboration success; which means that, behind all the technical complexities of implementing data science and AI initiatives, policy makers should always transmit the public value that the policy or programme ultimately pursues. Moreover, leveraging expert insight will ensure that alternative dimensions to problem solving are incorporated to ensure quality. Lastly, the different micro-management strategies that public managers should use in these ventures would then be contingent to their situational understanding, which has been referred to as sense-making.

Data accessibility

The article has no supporting data.

Authors' contributions

All authors contributed equally to all stages of study design and drafting the manuscript. All authors gave final approval for publication.

Competing interests

There are no competing interests.

Funding

The study is funded by HEFCE Catalyst Fund #E10, and the MINECO CSO2016-80823-P fund.