Considerations for development and use of AI in response to COVID-19

Highlights

• •Consider a CAIO as a key role to lead the use of AI in COVID-19 response.
• •Methods are needed to manage unpredictable, unexpected, or biased data.
• •Repurposed AI offers promise for rapid availability of applications.
• •AI accuracy rates achieved in development may be lower in real-life use.
• •Diverse AI team membership is meant to attain the best system performance.

Abstract

Artificial intelligence (AI) is playing a key supporting role in the fight against COVID-19 and perhaps will contribute to solutions quicker than we would otherwise achieve in many fields and applications. Since the outbreak of the pandemic, there has been an upsurge in the exploration and use of AI, and other data analytic tools, in a multitude of areas. This paper addresses some of the many considerations for managing the development and deployment of AI applications, including planning; unpredictable, unexpected, or biased results; repurposing; the importance of data; and diversity in AI team membership. We provide implications for research and for practice, according to each of the considerations. Finally we conclude that we need to plan and carefully consider the issues associated with the development and use of AI as we look for quick solutions.

1. Introduction

Virtually everyone across the world has been impacted by the COVID-19 pandemic originating from Wuhan China and wants quick solutions. Vint Cerf, vice president and Chief Internet Evangelist at Google and past Association for Computing Machinery (ACM) president, sees an “awesome responsibility!” as “our profession and the products it creates will have a prominent role in shaping our post−COVID-19 society” (Cerf, 2020, P. 7). Let’s hope we can quickly seize this responsibility. Indeed, the pandemic is expected to accelerate the rate and pace of technological change, particularly for the emerging artificial intelligence (AI) revolution (Wladawsky-Berger, 2020).

AI is seen as playing “a key supporting role in the fight to treat and stop” the virus and perhaps will “contribute to a solution coming faster than we would have otherwise” in the biotech field (Block, 2020, para. 31), among many other fields and applications. Since the outbreak of the pandemic, there has been an upsurge in the exploration and use of AI, and other data analytic tools, in a multitude of areas described by Naudé (2020a). For example, Landing AI, headquartered in Palo Alto, CA, USA, is developing a new AI system to monitor distance between individuals in the workplace (Sreeharsha, 2020). Researchers at Icahn School of Medicine at Mount Sinai in New York, USA developed a unique AI system that can rapidly diagnose COVID-19 based on a patient’s computed tomography (CT scans) of the chest, in combination with clinical symptoms, exposure history, and laboratory testing (Mei et al., 2020). BlueDot, an infectious disease outbreak software startup based in Toronto, Ontario, Canada, augmented traditional epidemiological models with machine learning to model the spread of the coronavirus in California to specific zip codes, using flight pattern data.

The rapid development and implementation of AI is viewed as “central to the COVID-19 picture” (Silverman, 2020, para. 3). AI is a broad field now encompassing areas such as robotics, natural language processing, vision and sensory systems, expert systems, and decision aids. Generally, AI refers to the ability of a machine to learn from experience, adjust to new inputs, and perform human-like tasks. There is no commonly accepted definition of AI because the definition has changed as this technology has evolved.

Business interest in AI has been increasing (Yapo & Weiss, 2018). “Each day seems to bring new instances of automation of routine tasks and the addition of artificial intelligence or machine learning algorithms to new domains” (Mueller, 2019, p. 1). The number of companies implementing AI-related technologies in the past four years has grown by 270 percent, according to research firm Gartner (Leprince-Ringuet, 2020). It is difficult to determine the proportion of businesses effectively using AI currently and to what extent. Some insight is provided by the results of a survey, of senior leaders at 30 Global 500 companies, which revealed that 30 percent of respondents are using AI for a selective range of functions and 17 percent were deploying the technology "at scale" within the enterprise (KPMG, 2019). This reveals that the majority of organizations may be unprepared for the rapid influx of AI-based applications related to COVID-19.

This paper addresses some of the many considerations for managing the development and deployment of AI applications, discussed in the context of the fight against COVID-19. We discuss the general necessity to align information technology (IT) strategy with business strategy in Section 2. In Section 3, we consider the unpredictable or unexpected occurrences in the data, and correspondingly the resulting unpredictable or unexpected results, including biased output. The repurposing of AI in an environment requiring rapid responses to find quick solutions is examined in Section 4. In Section 5, the importance of the role of data as the lifeblood in AI is highlighted. The critical necessity for diversity in team membership, for development and deployment, is emphasized in Section 6. We provide implications for research and for practice, in Sections 7 and 8 respectively, before the conclusion in Section 9.

2. The alignment of information technology strategy with business strategy

Key to managing any crisis is planning, preparation, and leadership. Primary in using any IT resources effectively, is the consideration of aligning IT strategy with business strategy to use IT resources effectively. Chief Information Officers (CIOs), and other senior executives, consistently agree that this alignment is among the top-most important IT management issues (Sabherwal, Sabherwal, Havakhor, & Steelman, 2019). Of course, IT does not drive strategy. Rather what is needed is strategic leadership to develop unique insight to solve a pressing need and then drive change to realize a vision. For example, at biotech start-up Insilico Medicine, based in Hong Kong, while the focus is still on using machine learning to find drug treatments for cancer, fibrosis, and many other diseases, the Chief Executive Officer (CEO) prioritized several projects related specifically to COVID-19, and their platform has been repurposed for virus research (Field, 2020).

For AI applications, companies may consider a Chief AI Officer (CAIO) to present opportunities for innovation which have the potential for dramatic impact, by contrast to the CIO who focuses on the delivery of IT services and the effect on firm performance. Davenport and Dasgupta (2019) recommend instituting competence centers (CC) or centers of excellence (COE) “to entrench AI” (p. 2). Coordinated, collaborative, centralized oversight of AI can be achieved with CCs or COEs, rather than taking a bottom-up approach to adopt AI applications, which is inefficient and unproductive (Davenport & Dasgupta, 2019). The CAIO is a natural position to head up such centers, which could house AI talent. Thus, the main responsibility of the CAIO is to identify areas appropriate for the application of AI, across the entire company, to add value in meeting the goals and objectives of the business strategy. Additionally, the CAIO will, in coordination with internal and external partners and vendors, manage development teams, oversee implementation of applications, and continuously monitor the performance of those applications.

3. Unpredictable, unexpected, and biased results of AI

AI can encounter unpredictable or unexpected occurrences in the data, and correspondingly generate unpredictable or unexpected results, including biased output, because of the complex nature of AI solutions. Bias in AI systems can “affect the brand value of the organization” (Gartner, 2020, p. 12). Thus, a consideration of bias as well as unpredictable or unexpected results in AI is essential. Machine learning-based systems currently in use may encounter vastly different data, reflecting the changed circumstances of life imposed by the coronavirus lockdowns (Heaven, 2020). For example, detecting fraud in credit card transactions relies on rules and predictive models to interpret a consumer’s spending pattern data and provide a fraud prevention payment score to approve the transaction. However, spending behavior has changed as people are not purchasing big-ticket items or spending in new places as they usually had done, requiring adjustments for the increase in purchases of garden equipment and power tools (Heaven, 2020).

Degree of confidence in AI results, or in data sets, may spill over to trust in the results. Level of familiarity with the use of AI may also impact trust in the recommendations made. Sufficient trust is necessary for an individual to act based upon the recommendations (Engler, 2020). What level of trust should patients and clinicians place in a system designed to predict which infected patients will become severely ill, to better prepare to deliver lifesaving medical treatment? In a 2019 survey, 2000 consumers in the US were asked “When you think about AI, which feelings best describe your emotions?” (Davenport, 2019). The results revealed the most common response (45 %) to be “Interested,” followed by “Concerned” (40.5 %), “Skeptical” (40.1 %), “Unsure” (39.1 %), and “Suspicious” (29.8 %). Davenport (2019) suggests that this deficiency of trust in AI is attributable to overhyping the claims regarding the capabilities of AI systems, a lack of transparency in revealing characteristics of the system and how it will be used, and a lack of external certification of the output produced. Engler (2020) regards the role AI is playing in response to COVID-19 to be overhyped in news article, in terms of tasks it performs, its effectiveness and scale, and disregard of the level of human involvement.

Biases may arise due to historical patterns of bias repeated in an AI application or learned by the application itself through machine learning. Specific instances of bias in machine learning algorithms have been reported (Yapo & Weiss, 2018), contrary to the claim that the neural network basis of machine learning affords “speed, accuracy and lack of bias” (Marr, 2016, emphasis added, para. 13). Since “AI has not yet been impactful against COVID-19” (Naudé, 2020b, p. 1), we present unrelated examples to show that bias can occur in the use of AI. Among the biases uncovered in AI applications are gender bias, in Google’s search (Carpenter, 2015) and speech recognition software (Larson, 2016); and racial bias in calculating scores to predict the likelihood of committing a crime (Angwin, Larson, Mattu, & Kirchner, 2016) and in face recognition software (Crawford, 2016).

According to Weyerer and Langer (2019), three stages contribute to AI-based bias: (1) AI input, (2) AI learning, and (3) AI outcome. Algorithmic bias appears to be the most frequently cited bias (Pandya, 2019). This bias arises from “how the software is designed, developed, deployed and the quality, integrity, and representativeness of the underlying data sources” (Pandya, 2019, para. 9). Data sets used to train neural network based systems can be inaccurate, biased, or incomplete. The neural network learns to recognize patterns in the data by weighting factors found within the data, thereby generating results that reflect imperfections within the data set. Often, historical hiring data from a company is used, with the intention of capturing attributes and traits of high-performing employees, which then may replicate prior institutional bias (Pandya, 2019). To mitigate or manage bias, it may be more important to achieve diversity in development and deployment teams, than in the training data set (Shellenbarger, 2019) because these teams could undertake an audit of the AI system for bias. AI itself could assist in screening for bias patterns in the characteristics selected to serve as attributes, in the data set, or in the model itself.

4. Repurposed AI for rapid responses to find quick solutions

Repurposed AI may be susceptible to producing unpredictable, unexpected, or biased results as the model was initially trained for the original purpose. Repurposed AI is not new, but may offer promise in an environment requiring rapid responses to find quick solutions. With a desire to find ways to speed up the development and implementation of promising COVID-19 applications, repurposing might be quicker because a similar existing application could be modified, rather than initiate the development of an entirely new system.

For example, consulting firm PwC developed an AI-based Automatic Contact Tracing tool, by repurposing an existing system initially used to track assets within a building (Sreeharsha, 2020). Indoor geolocation is used to track contact among employees in an office environment. Ambient signals, which are the constant background field of radio signals, are captured. Millions of these wireless signals are aggregated into a graph to analyze how these signals interact to determine the time and proximity of employee interactions at work. Information from this contact tracing system can be used to determine with whom an infected employee has had contact to reduce the risk of COVID-19 cases spreading within a company. Reports describing this system reported no details about the repurposing.

In another example, Smartvid.io, headquartered in Cambridge, MA, USA, offers an AI software product that assesses risk and safety at work sites for construction companies (Sreeharsha, 2020). This system uses a combination of cameras and various deep learning approaches for computer vision to analyze construction-focused images to look for specific indicators of risk in imagery. For example, is a worker on a ladder or working at heights? Aggregate data indicates the overall risk. Leveraging the capabilities of this software, Smartvid.io produced an AI-powered enterprise software product that detects if two individuals are closer than one-person length apart and if more than 10 individuals are gathered together. This system now also analyzes images to assist in enforcing social distancing on-site.

In one more example, Qure.ai Technologies Pvt. Ltd., a startup based in Mumbai, India, repurposed an AI system, originally developed to detect tuberculosis, to assist with diagnosing and assessing COVID-19 cases by checking computer tomography (CT) lung scans for signs and severity of the disease (Olson, 2020). A team led by the head of radiology, Francesco De Cobelli, at San Raffaele Hospital in Milan, Italy, immediately found the system made the disease look worse than it was. However, this AI application is in the early stages of development. Dr. De Cobelli’s team spoke daily with engineers from Qure.ai, who retrained the system using hundreds of X-rays of COVID-19 patients. Qure.ai CEO, Prashant Warier, said the early issues were resolved and that San Raffaele Hospital is now a paying customer. By comparison, it has taken five years to validate AI systems that can detect breast cancer. Skepticism remains as British radiologist and health-tech firm consultant Hugh Harvey commented, “There’s no way anyone has done that specifically for Covid in the past three months” (Olson, 2020, para. 6).

What can we learn from examples of repurposed AI? Little research has been done on repurposing information systems in general, or AI in particular. Mawson-Lee (2006) reported that a system designed to manage all requests made to the IT unit within a company, “can be easily repurposed… for a wide range of task management purposes” (p. 79), but with no explanation of considerations in doing so. Regarding repurposing of Russell, Dewey, and Tegmark (2015) note the necessity to maintain control over system training and use. They argue that in repurposing, if that system is “selecting the actions that best allow it to complete a given task, then avoiding conditions that prevent the system from continuing to pursue the task is a natural subgoal” (p. 111). AI systems which can be controlled during repurposing are termed “corrigible” (Soares, Fallenstein, Yudkowsky, & Armstrong, 2015). Santoni de Sio and van den Hoven (2018) identify two requirements for an autonomous system to remain under human control. The first, termed a “tracking” condition, specifies that the system should respond to both moral reasons for designing and deploying the system and relevant facts in the environment. The second, termed a “tracing” condition, stipulates that the system be designed to trace back the outcome of its operations. Bergstein (2020) notes that AI systems can be confused by new situations, inherent in being trained for one task and being taught to perform a different task. However in this repurposing process, the system may not retain all of the expertise gained in performing the original task, which is termed “catastrophic forgiving.”

5. Importance of data

AI uses vast quantities of historical data, with known outcomes, to discern patterns in that data and make predictions about the future. The accuracy of those predictions depends upon the quality of the data set used to train the model, the quality of the data set used to produce the prediction, the design of the algorithm, and the performance of the system using real-world data (Silverman, 2020). In real-world settings, the AI system must be trustworthy in its performance, which depends upon the data. For example, researchers have expressed skepticism about the real-world performance of models, developed to detect COVID-19 in the lungs, touted by developers as attaining a 90–96 percent accuracy rate (the percent of predictions correct during development), because the training data set is not representative of actual patients (Council, 2020; Olson, 2020). The amount of data and the diversity of the populations it represents help boost the accuracy of the system (Council, 2020). Nonetheless, 96 percent accuracy “is suspiciously high for any machine learning problem” and real-world deployment nearly always degrades system performance (Engler, 2020, para. 10).

Wide-scale access to relevant big data is key. If training data is not sufficient and relevant, the output produced will be inferior or incorrect. Rather than repeat the old adage “Garbage in, garbage out,” we instead agree that “There’s another law of artificial intelligence and machine learning: The more data, the more accurate the results will be” (Kessler, 2020, para. 8). A major barrier to success with AI is a lack of good data and a major risk to organizations is not understanding the context of that data (Zielinski, 2017). The selection of the data set should be substantiated for its intended use in the model; and the quality and relevance of the data set should be subject to rigorous assessment (Board of Governors of the Federal Reserve System., 2011). Specifically, training data must be vetted for quality, integrity, and representativeness in terms of diverse characteristics of the underlying data sources (Pandya, 2019) as well as accuracy, relevance, and freshness (Roselli, Matthews, & Talagala, 2019). Data used in production processing may differ from the training data set. Thus, production data should be monitored for completeness, the expected distribution anticipated, uncommon cases, and timeliness (Roselli et al., 2019).

Much of the data being produced about COVID-19 is compiled in separate silos by the government, academia, and business (News Bites - Private Companies, 2020). With frequent new discoveries made about the virus, real-time updates must be made to the data and to the models (Council, 2020). A cohesive data strategy, that lacks silos and enables deep learning technologies to create new insights, is necessary.

An example of an effort to provide for integrated data access is that of C3.ai, a leading enterprise AI software provider headquartered in Redwood City, CA, USA, which produced a vast open cloud-based repository, known as a data lake (Kessler, 2020), A variety of large data sets from a multitude of different sources (see C3.ai, 2020) is provided, including location and confirmed case data from the World Health Organization and Johns Hopkins, genome sequences of COVID-19 samples from the National Center for Biotechnology Information Virus Database, global patient data such as symptoms and lab results, and the Milken Institute’s database of treatment and vaccine trackers (News Bites - Private Companies, 2020). The C3.ai COVID-19 Data Lake, one of the largest integrated sources of COVID-19 data in the world, is publicly available, at no cost, to global research and scientific communities (C3.ai, 2020). Pre-established linkages enable researchers to easily navigate and explore all of the associations within and across the unified data sources, which are updated in real-time. The CEO of C3.ai, Tom Siebel, commented “This enables scientists to perform very advanced research using AI, accurately predict the spread of the disease, and evaluate the efficacy of social mitigation” (News Bites - Private Companies, 2020, para. 4).

The use of data requires a consideration of data ethics and regulations, especially within the context of healthcare. Though the considerations are relatively new and vary from source to source, some principles are universal (The Institute of Internal Auditors, 2020). Regarding ownership and consent, it is individuals who own and therefore grant consent for the use of their personal data. Once consent is provided, transparency and disclosure about how the data is used must be provided. Privacy and security should be maintained through all reasonable efforts. In terms of currency, individual should be made aware of any financial transactions resulting from the use of their personal data. Compliance with regulations which govern and control the use of personal information must be maintained.

6. Diverse teams for development and deployment

A diversity of disciplines is the key to success in AI, especially to minimize risk associated with rapid deployment of new technologies to respond to the coronavirus (Gartner, 2020). Such teams enable alignment of “a company’s culture, structure, and ways of working to support broad AI adoption” (Fountaine, McCarthy, & Saleh, 2019, p. 64). Team membership should be well-rounded, from a wide range of backgrounds and skill sets, for complex problem-solving with innovative solutions and for recognizing the potential for bias. To address issues such as bias, ethics, and compliance, among others, roles such as an “AI Ethicist,” attorney, and/or review board, may be added (Davenport & Dasgupta, 2019, p. 5). “The biases that are implicit in one team member are clear to, and avoided by, another,… So it's really key to get people who aren't alike" (Shellenbarger, 2019, para. 10). Engler (2020) underscores the importance of subject-matter experts, especially for COVID-19 applications.

7. Implications for research

In our discussion, we highlighted the considerations for developing and deploying AI in the context of COVID-19 related applications. For these considerations we have formulated research questions.

7.1. Future research addressing the alignment of IT strategy with business strategy

In aligning IT strategy with business strategy, future research could evaluate actual common crisis-management successes and failures applicable to planning for and responding to pandemics. The results could be applied to the formulation of a conceptual framework framed specifically around pandemics that can be used to assist in strategy formulation.

7.2. Future research considering unpredictable, unexpected, and biased results of AI

There is very limited academic research focusing on understanding the impact of the use of AI applications applying rigorous methods and theories (Duan, Edwards, & Dwivedi, 2019). With the potential for unpredictable, unexpected, and biased results in an environment within which development and deployment is hastened by necessity, future research to increase this understanding becomes critical.

AI can exacerbate existing biases and create new biases in judgment decision making (Angwin et al., 2016; Carpenter, 2015; Crawford, 2016; Larson, 2016; Marr, 2016; Yapo & Weiss, 2018). Biases may arise due to biases encoded in an AI application itself, learned by the application through machine learning, or introduced by the users. Methods with which to mitigate them could be explored. For example, level of trust in cognitive technologies such as AI and machine learning, and their ability to produce the same judgment as human decision makers, may lead to bias. Degree of confidence in AI or in data gathered may spill over to confidence in interpreting the results. Level of familiarity with the use of AI may impact bias. If biases persist or emerge, new methods to mitigate bias may have to be developed. For example, mechanisms to mitigate bias may be built into the AI application itself, such as ensuring algorithm transparency by providing an explanation of the results or decision recommended.

7.3. Future research on repurposing AI to find quick solutions

Research on repurposing AI is clearly needed as it is an approach being used to hasten development during the COVID-19 pandemic, but within which little research has been done. Such research might yield insights beneficial for future pandemics and other crises requiring quick solutions. Similarities between the specific purposes of the original and repurposed systems could be explored to determine when this development approach is appropriate. The development techniques useful in repurposing could be defined and documented. The impact of repurposing on bias could be explored to discover whether new biases are created due to the change away from the original purpose. Additional research could further explore “corrigible” AI systems, “tracking” conditions, “tracing” conditions, and “catastrophic forgiving,” among other characteristics.

7.4. Future research on data for AI

Extensive research has been done on data quality focusing on the reach, solutions, and methods for data quality assessment (Madnick, Wang, Lee, & Zhu, 2009). However, little research examines the actual level of data quality within organizations (Nagle, Redman, & Sammon, 2020). We concur with the dire necessity to assure data quality, especially in AI applications to support the healthcare industry. Nagle et al. (2020) propose a typology of data quality which could serve as a useful starting point to formulate preemptive actions companies could take against the most frequent types of data error.

7.5. Future research on diversity in AI team membership

Previous research has examined diversity in AI teams by considering whether the focus should be on the diversity of the team or on the strength of each member, and whether a team of diverse and weak members can outperform a uniform team of strong members (Marcolino, Jiang, & Tambe, 2013). Such questions are particularly relevant when development and deployment efforts are to be undertaken quickly. Further, Shellenbarger (2019) encourages diversity in the development team to mitigate or manage bias in AI systems, which may be particularly relevant to quickly built repurposed AI. Future research could examine the contributions and capabilities of both development and deployment team members according to the specific purpose of the AI system.

8. Implications for practice

Companies are expected to ramp up automation, including AI application development, in the aftermath of the coronavirus outbreak (Loten, 2020) and in doing so, they must learn quicker than ever before (Wladawsky-Berger, 2020). AI is playing a key supporting role and key in using AI is thoughtful and informed planning and preparation which addresses many considerations.

8.1. Implications of aligning IT strategy with business strategy

Implementation requires leadership. Companies may consider a CAIO as the key role to determine how resources devoted to AI might be used for advantage in realizing business strategy. In coordination with CCs or COEs and/or internal and external partners and vendors, the organizational response can be rolled out and monitored by the CAIO.

8.2. Implications of unpredictable, unexpected, and biased results of AI

Mechanisms for contending with unpredictable, unexpected, or biased data, and the potential for such results must be in place, especially in the case of repurposed systems. Remnants from the system’s original purpose may remain, causing incorrect results.

8.3. Implications of repurposing AI to find quick solutions

Among the considerations addressed, it is repurposed AI which offers promise for rapid availability in real-world application. However, as previously cautioned, these systems must be fully tested for the new use before being deployed. Historical patterns of bias may be repeated in the repurposed system or new biases may arise. Loss of confidence in AI results may spill over to loss of trust in the results, which could ultimately lead to disuse of such an expensive resource.

8.4. Implications of data for AI

An AI system should be used with caution because the accuracy rate attained during development may not be attained in real-life situations. Hence, wide-scale access to relevant quality data is key to thoroughly train the system before it is deployed. Additionally, data ethics and regulations, especially within the context of healthcare, must be addressed.

8.5. Implications of diversity in AI team membership

The goal of diverse AI team membership is to attain the best possible performance of the system in use, in compliance with data ethics and regulations. Thus, it is team member knowledge which is the focus in team member selection.

9. Conclusions

The goal of this paper is to provide considerations for organizations in the development and use of AI systems for COVID-19, and any future pandemics. AI has a role in assisting with many relevant questions, including diagnosis, contact tracing, social distancing, workplace safety, and more. Many considerations must be addressed for effective use in real-world application to ensure the solutions meet the needs of the business as mistakes could slow progress in dealing with the virus. Included among the many considerations are aligning information technology (IT) strategy with business strategy; considering unpredictable or unexpected results, including biased output; repurposing AI to find quick solutions; the importance of the role of data as the lifeblood in AI; and the critical necessity for diversity in team membership. Among the considerations addressed, it is repurposed AI which has received little research attention but offers promise for rapid availability of systems in real-world application. We hope for success, in this approach and others, in the quest for AI systems to benefit humanity as we respond to COVID-19 and beyond.