A Framework for Augmented intelligence in Allergy & Immunology Practice and Research: A Work Group Report of the AAAAI Health Informatics, Technology & Education Committee

Abstract

Artificial and augmented intelligence (AI) and machine learning (ML) methods are expanding into the healthcare space. Big data is increasingly used in patient care applications, diagnostics, and treatment decisions in Allergy & Immunology. How these technologies will be evaluated, approved and assessed for their impact is an important consideration for researchers and practitioners alike. With the potential of ML, deep learning, natural language processing (NLP) and other assistive methods to re-define healthcare usage, a scaffold for the impact of AI technology on research and patient care in Allergy & Immunology is needed. An AAAAI Health Information Technology and Education (HITE) subcommittee workgroup was convened to perform a scoping review of AI within healthcare as well as the specialty of Allergy & Immunology to address impacts on Allergy & Immunology practice and research as well as potential challenges including education, AI governance, ethical and equity considerations, and potential opportunities for the specialty. There are numerous potential clinical applications of AI in Allergy & Immunology that range from disease diagnosis to multi-dimensional data reduction in electronic health records or immunologic datasets. For appropriate application and interpretation of AI, specialists should be involved in the design, validation and implementation of AI in Allergy & Immunology. Challenges include incorporation of data science and bioinformatics into training of future Allergists Immunologists.

Introduction

Artificial intelligence, augmented intelligence (AI) and machine learning (ML) are gaining traction in healthcare with the promise of providing assistance to clinicians in interpreting complex datasets, improving disease diagnosis, and facilitating clinical decision support(1). AI enables computers to imitate human intelligence with its ability to observe, problem solve, and learn whereas ML and the related area of deep learning (DL), is the ability of systems to learn, extract patterns, and refine performance over time(2). Augmented intelligence leverages machine learning to enable actionable use of data in contrast to autonomous replacement of human intellect. For example, artificial neural networks (ANN) are a type of DL model which utilizes a series of layers to analyze data inputs for prediction or regression via a nodal structure that mimics the human brain(3). These and other DL algorithms allow for analysis of high-dimensional and complex data which is being used widely in research settings.

The mechanics of AI center on analysis of machine-readable elements assembled for the purpose of predicting an outcome (i.e., classification or diagnosis) of interest(4). The use of AI algorithms in healthcare depends upon the construction of validated datasets derived from structured and unstructured data of relevance(5). If validated datasets are maintained with appropriate governance, they may be amenable for algorithmic mining and model development.

Expenditure on healthcare AI is projected to increase from $2B in 2018 to $34B by 2025(6). The National Academy of Medicine has emphasized the “quadruple aim” of improved outcomes, reducing cost, and improving patient and physician/provider experience in healthcare delivery. AI has the potential to positively impact these aims (2) while advancing professional development and medical education (Figure 1).

Figure 1.

Quadruple Aim of augmented intelligence applied to allergic and immunologic disorders. Abbr: EHR-electronic health record, PID-primary immunodeficiency or inborn errors of immunity.

Specialties such as cardiology, oncology, and radiology were early AI adopters whereas Allergy & Immunology is beginning to incorporate the use of AI. AI and ML appear to be well suited for the field of Allergy & Immunology where large, high-dimensional datasets are common. The use of big data approaches for infectious disease outbreak(7) prediction, asthma tracking(8,9), and immunologic modeling from vaccination(10) are contemporary examples of use of AI. Similar to the push for adoption of evidence-based medicine in medical education, awareness of the data sources being utilized in models, application to clinical decision support (CDS), assessment of quality, transparency, reproducibility and transferability are new areas for trainee education(11).

This AAAAI workgroup report aims to develop a framework for understanding specialty-specific issues relevant to AI. Using search terms and Medical Subject Headings (MeSH across the AI, ML, data science, data governance and systems biology landscape along with common allergic conditions (e.g “asthma”, “food allergy”, “drug allergy”, “immunodeficiency”, “atopic dermatitis”, literature was reviewed via PubMed to understand the current status, and potential impact of AI on clinical practice and research in Allergy & Immunology. Readers should note that the AAAAI does not have a formal position on AI use and implementation at this time and this WG report is a summary of recent and relevant literature across the AI spectrum as assembled by membership of the AAAAI Health Informatics, Technology & Education Committee (HITE). In addition, opportunities for utilizing AI, exploring potential areas for standardization including operational difficulties that might impact adoption of AI in Allergy & Immunology will be explored. We will delve into impacts of AI on research and the need for development of a framework for AI in Allergy & Immunology prior to broad adoption or implementation.

Impact on Allergy & Immunology Clinical Practice

The clinical practice of Allergy & Immunology encompasses a variety of healthcare practitioners such as physicians, allied-health practitioners, nurses, as well as researchers, and laboratory technologists. Additionally, policy makers and payers intersect with health technology such as electronic records, telemonitoring interact to create a complex network of stakeholders involved in assuring patient safety and quality, appropriate and timely diagnosis of diseases, and appropriate selection of treatments. Because the immune system is extraordinarily complex, this intersectionality is not well addressed in the current healthcare system. Although AI seems well-poised to address some of the challenges, there are as yet several limitations on its implementation and use. In the next section we will highlight some of the nascent and potential applications of AI that are applicable to Allergy & Immunology research and practice.

Disease Diagnosis

Healthcare datasets, such as electronic health records (EHRs), and pathology images (for e.g. eosinophils in eosinophilic gastrointestinal disease biopsies) present unique challenges and opportunities for data extraction(6). A major focus of AI in healthcare is automated disease detection(12) where electronic phenotyping (EP) of patients with distinct clinical features can enable machine-readable information and detection(13). Development of precise EPs has proven useful for searching and ascertainment of large datasets to identify disease entities(14). Similar approaches could be applied for Allergic and Immunologic diseases and will be discussed in this AAAAI workgroup report.

Within the field of Allergy & Immunology, an unmet need includes detection of patients at risk for Primary Immunodeficiency disorders (PID) or Inborn Errors of Immunity (IEI) prior to fatal infection or organ damage. Claims data has been modeled for disease detection as one approach, and ML has been applied to available laboratory data such as calculated globulin fraction to detect hypogammaglobulinemia(15,16). Additionally, diagnoses from EHRs have been used to determine risk and provide guidance about the most likely International Union of Immunological Societies (IUIS) category of PID(17). These examples of structured health data mining show the possibilities of applying AI systems within learning health systems to improve diagnostic rates for patients with rare disease within the context of Allergy & Immunology. Figure 2 illustrates how undiagnosed PIDs or IEI’s benefit from iteratively updated clinical data with ongoing disease characterization. Data extracted and housed in a digital learning repository may be processed and analyzed with a resultant AI algorithm that assists with more accurate disease diagnosis for future cases.

Figure 2.

Augmented intelligence (AI) workflow in a learning health system

Clinical Practice and Diagnostic Decision Support

Allergy & Immunology patients may present with significant complexity requiring analysis of both structured and unstructured EHR data for optimal information use. Natural language processing (NLP) is an AI subdomain that may be applied to Allergy & Immunology through extraction of unstructured concepts (e.g. disease symptoms) from free text entered in clinical notes. NLP is proposed as a mechanism for predicting patient outcomes(e.g. exacerbations or hospitalization for asthma) that enable targeted interventions for improving outcomes in at-risk individuals. ML and NLP models have been built using EHR data and remote monitoring to classify asthma severity using features such as medications, symptoms, lung function, and comorbid conditions(18-20). In addition, some ML models have predicted asthma exacerbations from telemonitoring data in adult patients requiring ER evaluation and hospital admission with high sensitivity, specificity and accuracy(21,22). Further validation efforts using balanced population data will be required to further validate these data. The ability to aggregate data into AI models from various data streams (e.g., EHR, claims data, wearables, and immunologic data) remains a challenge; however, two recent studies indicate that it may be possible(23,24) to use AI models for asthma prediction. In the context of asthma, EHR extracted data such as peak expiratory flow rate, long-acting beta agonist and inhaled corticosteroid use are processed and analyzed for asthma prediction and assessment of disease control. (Figure 3). The quality, size and diversity of the datasets influence accuracy and scalability to diverse patient populations. Furthermore, there is an increasing number of stakeholders in AI, and the liability of clinical decision software vendors and practitioners remain unresolved(25). Beyond asthma, many other diseases may benefit from assistive technologies including drug allergy, IEI and immunodysregulatory disorders, and food allergy(Table 1).

Figure 3.

Recent examples of machine learning (ML) approaches for asthma prediction and control. AI-augmented intelligence; EHR-electronic health record, PEFR-peak expiratory flow rate, LABA-long-acting beta-agonist, ICS-inhaled corticosteroid, EHR-electronic health record.

Table 1.

Potential Clinical and Laboratory Applications of AI in Allergy & Immunology

Area of Allergy & Immunology	Unmet Need	Potential application of AI
Inborn errors of Immunity/Immune dysregulatory disorders	Early diagnosis; phenotyping; prognosis	Mining EHR Information & Classification
Adverse reactions to drugs or vaccines	Identifying causative drug; alternate drug tolerance prediction Identifying reaction type	Timeseries analysis/deep Learning Phenotyping
Asthma	Prediction of exacerbations, endotypes, and response to therapy Biomarker identification	Timeseries analysis/deep Learning Unsupervised learning
Food allergy	Biomarker Identification for prognosis, tolerance	Unsupervised Learning
Eosinophilia and EGIDs	Diagnosis; phenotyping; endotyping; prognosis prediction; response to therapy	Mining EHR information & phenotyping
COVID19 related disease	MIS-C prediction Contact tracing/disease spread prediction	Mining EHR information & classification Clustering/geo-spatial analysis
Laboratory immunology	Biomarker identification Disease correlations	Unsupervised learning Deep learning
Drug discovery	Drug repositioning	Deep learning, Support vector machines

Abbr: EGID-Eosinophilic Gastrointestinal Syndromes; EHR-Electronic Health Records; MIS-C-Multisystem Inflammatory Syndrome-COVID

Drug Allergy and Toxicity

The use of ML to supplement CDS for clinical applications in real time is particularly appealing for applications such as best practice alerts and smart order sets. CDS can be used to identify patients at higher risk for allergic diseases and/or to predict the optimal choice of drug as influenced by patient characteristic, drug-drug interactions and available formulary choice. ML models showed superior predictive accuracy as compared to logistic regression in predicting patients at future risk of beta-lactam allergic reactions in adults(9,26). Some studies used predictive modeling for non-IgE mediated side effects such as drug-induced hepatotoxicity using molecular structure, lipophilicity and other pharmacokinetic characteristics(27,28). These studies used published datasets available through the US FDA Adverse Reporting System (FAERS), the European Union Adverse Drug Reaction reports, and the Observational Medical Outcomes Partnership (OMOP) (29). As datasets for molecular descriptors and toxicogenomics increase, ML models such as ANN (Figure 4) may be used to predict adverse drug effects or risk for allergy in drug development, research(3), and post-marketing surveillance.

Figure 4.

A beta-lactam artificial neural network to predict risk of reactions. Abbr: IgE-immunoglobulin E; PCN-penicillin. This neural network takes parameters (e.g. patient age, duration or location of drug rash, suspected culprit medication, etc.), represents them as numbers, and uses them as “inputs” to the neural network. Non-linear mathematical formulas are used to pass the information between the nodes in the hidden layers of the neural network, ultimately resulting in an “output prediction” for a given set of input parameters. In the case of a drug allergy prediction algorithm as shown, the output could be a probability of high, medium or low risk of developing a reaction (e.g. p >0.8 = high risk; p <0.20 = low risk and between is medium). Neural networks are initially “trained” using data for which the output is known. For example, it may be a data set containing information about a large group of patients who have been evaluated and challenged for penicillin allergy. The neural network takes this known data, passes the inputs through the model and “tunes” the mathematical formulas in the hidden layers to maximize the prediction accuracy of the model. Then a new set of parameters (eg, for a new patient being evaluated for drug allergy) is input into the model to generate a prediction of the likelihood of PCN allergy.

AI applications to COVID Immune Responses

AI models were vital for synthesizing vast amounts of data generated in the COVID-19 pandemic. Classifying the SARS-COV-2 genome and scaling vaccine production are two of the approaches being advanced by AI(7,30). Use of AI and data science helped understand patterns of viral spread and which communities may be most vulnerable(31). Additionally an unsupervised learning approach called T-REX (Tracking Responders Expanding) can characterize lymphocyte subpopulations to provide early signs that herald diagnosis and enable insights into host defense mechanisms(32). Understanding immunologic and/or vaccine responses in immunocompromised or immunodeficient patients remains an unmet need. Some are attempting use of AI/ML for multi-criteria decision-making (MCDM) to identify patients in most need of convalescent plasma(7). ML approaches are being leveraged to classify immune cell stress features and predict antiviral responses for drug repurposing efforts(33,34). Lastly, ML is being coupled with reverse vaccinology to predict viral-host protein interactions and identify druggable targets and vaccine epitopes(35).

Practice Management

The practice and management of medicine is a target for automating and improving efficient care delivery. The administrative cost and adverse effects of EHR clinical workflow have increased(9, 10) and have led to clinician burnout(36,37). Although EHRs have served regulatory, billing, and revenue needs, prioritizing the patient-clinician interaction and user experience are aspects of the Quadruple Aim (Figure 1). Practice management could be significantly streamlined with AI. Potential uses include clinical order set automation(13) where NLP evaluates clinical entries to ascertain drug reactions (14,15), and utilizes speech-to-text for clinical documentation(38,39). More companies are developing digital products such as “chatbots”, or automated support chat features, for patient triage. Digital scribes employ NLP and speech recognition with EHR integration for outpatient clinical encounters. Other potential applications for practice management may include AI workflows for prior authorizations, billing, supply chain of extracts or vaccines, and optimizing patient triage or clinic flow(39). AI could extract data from the EHR to streamline order sets for medication prescribing, diagnostic testing and referrals. However, engagement of clinicians in the development of AI tools and operationalization of AI in the EHR will be crucial. Future research funding of AI in healthcare should focus on the impact of AI on clinician workflow, physician well-being and clinical outcomes.

Analysis of immunologic data

During the last decade, the field of Immunology has seen an increase in the volume and complexity of biologic data that can be generated, largely driven by advances in multiplexed technologies, imaging, and genomic sequencing(40). Frequently referred to as multi-“omics”, these data include DNA sequence, gene expression/RNA sequence, and assessment of proteins, lipids, metabolites, the microbiome, and environmental/lifestyle exposures (Table 2). As each data type can capture different attributes(41), complementary and integrated approaches for immunological investigation can be employed. In order to process and transform such large-scale, high dimensional data into biological and clinical insights, computational tools, including those that apply AI and ML approaches, have been implemented in both basic and clinical immunology.

Table 2.

Clinical and Research AI applications in “omics” in Allergy & Immunology

Data type	Information generated	Assay types	Clinical or research application in Allergy & Immunology
Genomics	Genetic sequence, to include SNPs, rearranged TCR/BCR sequences, open regions of chromatin, DNA methylation status, and the microbiome	Next-generation sequencing, microarray	Association between the methylation status of mononuclear cells and risk of asthma development
Transcriptomics	Sequence of expressed RNA transcripts, including mRNA and rRNA (often used in microbiome analysis)	Next-generation sequencing, microarray	Comparison between the skin and gut microbiome
Proteomics	Identification and quantification of proteins in biological samples	Mass spectrometry NMR spectroscopy Affinity-based methods	Evaluation of serum proteins to identify anaphylaxis phenotypes and associated protein signatures
Lipidomics	Identification and quantification of lipids in biological samples	Mass spectrometry NMR spectroscopy	Lipid profiles in 3rd trimester pregnant women and risk of infant atopic disease
Metabolomics	Identification and quantification of metabolites in biological samples	Mass spectrometry NMR spectroscopy	Association between serum metabolites and lung function in children with severe asthma
Phenomics	Clinical diagnoses, laboratory and radiology results in a format that can be queried	EHR data Clinical laboratory, radiographic data	Identification of trends in office PFTs and rescue inhaler use to predict likelihood of future severe asthma exacerbations
Exposomics	Exposures experienced from conception to death. Includes diet, lifestyle, climate, environmental, and occupational exposures	Questionnaires, biomonitoring data, environmental or dwelling measurements	Identification of patterns between local particulate matter concentrations, local elementary school ventilation systems, and asthma exacerbations in urban areas.
Cell profiling	Proteins expressed on the inside and outside of cells	Flow cytometry Mass cytometry/CyTOF	Use of mass cytometry to identify unique populations of cells found in milk-allergic children but not healthy age-matched controls.

Abbr: BCR-B-Cell Receptor; CyTOF-Cy time-of-flight; DNA-deoxyribonucleic acid; EHR-electronic health record; mRNA-messenger ribonucleic acid; rRNA-ribosomal RNA; NMR-nuclear magnetic resonance; SNP-single nucleotide polymorphism; TCR-T-cell receptor; References: (43,87,88)

While the use of ML in common clinical practice is still nascent, researchers have already successfully developed ML tools to make significant contributions in research. So far, ML methods have been used to analyze genomics data to understand pathophysiology, define biomarkers, and dissect endotypes of allergic diseases such as asthma, atopic dermatitis and food allergy (reviewed in (42,43)). Similar approaches are being employed in clinical immunology including use of support vector machine (SVM) for candidate gene identification in IEI(44), and ML algorithms to analyze high dimensional mass cytometry data to identify lymphocyte subpopulations for influenza vaccine response prediction(10).

As ML methods are increasing in the medical literature, providers must understand some of their limitations. A common pitfall is the use of data inputs of varying quality. When the data contains artifactual variations or is not representative of heterogeneity in human samples, ML algorithms can mis-interpret the dataset(45,46). Although several groups have developed AI algorithms for flow cytometry to detect specific lymphocyte subpopulations(45), the differential process of fluorescent labelling, cytometer voltage selection, and gating strategies to define specific cell subpopulations are operator dependent, lack standardization, and have a high degree of variability such that one cell subpopulation may be defined differently by disparate groups(45,46). These variations can affect model generalizability and performance. As such, application of ML to “real-world” biological samples requires standardizing data analysis prior to broad scale implementation. Other common pitfalls are reviewed in detail elsewhere(47).

Rapid technological advances allow biological characterization at increasingly high resolution and sensitivity. Most recently, single cell profiling has accelerated the ability to identify and phenotype cell types (48). For example, single cell RNAseq (scRNAseq) can identify CD4+ T-cells enriched in endoscopic biopsies of patients with active eosinophilic esophagitis (EoE) when compared to EoE in remission(49). In addition, single molecule array digital ELISA technology now allows ultrasensitive proteomic quantification, which has been used to evaluate plasma concentrations of interferon-α in patients with polygenic and monogenic interferonopathies(50).

The ability to generate high-dimensional data using a variety of modalities allows for more comprehensive biological characterization. In AI, for example, the Mechanisms of the Development of ALLergy (MeDALL) project applied ML methods to several classes of omics data (including transcriptomics, proteomics, DNA methylation and GWAS data) to identify two phenotypes of IgE-associated disease (“monosensitization” vs “polysensitization”) and found that those with polysensitization were more likely to have severe and persistent allergic disease(51) (recently reviewed in(43)). However, analysis of multi-omic data requires computational and mathematical approaches that can process and model hundreds of thousands of measured parameters and extract those that are relevant to key biological variables. Some of the issues that come into play include: 1) evaluating the data for differences that arise from batch effects; 2) adjusting for multiple comparisons, 3) data reduction (e.g., principal component analysis); and 4) determining the optimal ML model for analysis (e.g., SVM or neural network) (52). Readers of studies using multi-omic data should consider these and related issues when evaluating study quality and subsequent conclusions.

Candidate Selection for Clinical Trials

Phenotyping of clinical notes and EHR data to identify distinct clinical concepts became highly visible with the advent of the federally funded Electronic Medical Records and Genomics (eMERGE) program(53) and other efforts within the bioinformatics community(54). Applications such as clinical research cohort selection(55) and clinical phenotyping(56) became early and visible applications of AI. NLP was identified as an approach for overcoming clinical trial recruitment hurdles by using EHR data to identify potential research participants, where classifying asthma phenotypes(8) or atopic dermatitis severity(57) can enable trial recruitment. Similarly, mining health record information in other practice settings, such as emergency department unstructured allergy data(58), could be key to intelligent extraction of data across the care continuum. In a recent publication, Seol et al. trained and validated a custom NLP algorithm from an Olmsted County Birth Cohort(59) and used predetermined asthma criteria to create an ‘asthma predictive index’ to distinguish patients predicted to have asthma. Further optimizing tools such as this and developing algorithms for other disease entities, could broadly improve clinical trial candidate ascertainment.

Challenges for Al

Data Procurement and Modeling

There are many potential data considerations that remain as challenges for AI when applied to the Allergy & Immunology context. The end-to-end AI development process involves extracting appropriate data for analysis, addressing key elements of data quality, and data provenance which is a transparent description of where the data comes from. Many have indicated the need for a reference card, or so-called “model cards” (60) that document relevant performance characteristics and intended use of a model. From a purely data perspective, data inputs, use-cases, and data procurement are key factors that directly impact the models that are built. Governance to ensure optimal model inputs, best practice algorithmic model development (i.e. training, validation and testing), and finally operationalizing the model effectively which includes a sustainment plan for tracking longitudinal performance. Lastly model deployment requires regulatory oversight to assure ethical and equity considerations are met as well as to optimize reproducibility of AI. The details of a granular end-to-end analytics pipeline is beyond the scope of this manuscript; however, we will discuss key elements that are widely publicized and relate to concerns about fairness and reproducibility in digital medicine(61,62).

Ethics and Equity Implications

Ethical considerations must be considered at all stages of AI algorithm development but are particularly important in the stages of data procurement and model development. As AI should enhance the healthcare of individuals and populations, bias in algorithms is a critical concern. One study analyzed a commercial prediction algorithm to guide health decisions for US patients. Using healthcare utilization and spending on care, the algorithm falsely attributed better health scores to Black patients compared to sick White patients despite the fact that unequal access to care and healthcare disparities likely resulted in algorithmic bias(63).

Patient safety and health equity using AI are crucial concerns. Healthcare practitioners must incorporate informed consent regarding diagnostic error, accountability, privacy and cybersecurity considerations(1) when AI technology is utilized in their care. AI-enabled misdiagnosis is also a concern, particularly when relying solely on CDS or if inaccurate data or homogenous populations were used to train an algorithm. Deep neural networks (DNNs) may be a “black box” for clinicians, (1) and over-reliance on AI may make it difficult to counter a clinical decision suggested by the AI(64). Guidelines on how to override algorithms must be considered. Similarly, there will need to be a mechanism for reporting of safety events including misdiagnoses or inappropriate treatment decisions. Systems to address areas of conflict between ML CDS tools and clinician judgment will need to be developed. In addition, medicolegal aspects of use of (or inability to access) AI technology for clinicians in different practice settings will need to be addressed. Engaging frontline users in the adoption of AI will be important in future applications.

There is substantial concern regarding health equity in AI. While clinicians are not immune to racial and other biases, AI systems’ ability to achieve massive scaling means these biases could be amplified more than by humans alone. In addition, some clinical trials remain insufficiently diverse in their recruitment(65); as such, datasets may result in inadequate assessments for underrepresented populations(64). Moreover, consideration of data inputs (such as healthcare utilization) may be an ineffective proxy for illness(63), and health inequities may be exacerbated when bias is “baked-in” to AI algorithms(60). On the other hand, disparities of patient experience of pain may be re-examined using ML as was performed in a study of knee pain(66). In this study, an algorithmic assessment of radiographic features using a diverse patient training set showed less pain disparities by race than the radiographic scale currently used in routine practice. Since radiographic severity may impact when arthroplasty is offered to patients, ML has the potential to alter future outcomes. Clearly, both possibilities exist with AI having the potential to worsen bias, but also have the potential to improve understanding of diverse health outcomes and improve care for historically disadvantaged patients.

AI Deployment

AI Governance

Al clinical applications will likely be accelerated by the proposed regulatory framework of “software as a medical device”(SaMD; (67)). Here, SaMD is defined by the FDA as a medical device platform or virtual network that is used for medical purposes but is distinct from hardware with embedded software. Numerous consensus documents exist related to ethical AI use. The FDA and UNICEF have implemented ML “best practices” and operational standards which enable fairness in respect to the proprietary nature of the technology and mitigate bias(68,69). With this broad perspective for both AI impact and potential for bias, governance teams must have diverse stakeholder representation and “first do no algorithmic harm” with an intent to optimally balance utility and fairness (68) in the process of assuring algorithmic equity. To facilitate optimal development and operational practices, thought leaders and broad stakeholder groups have begun to emerge. For example, the Partnership on AI formed “About ML” which focuses on transparent ML lifecycles to improve responsible scaling of ML and define optimal ML lifecycles(70). Although the “black box” of AI may not be fully reducible, and alternate statistical methods may be used, the Partnership on AI is focused on engaging multiple stakeholders to make AI more equitable.

Training and Education

Currently, AI as a field is outside of the scope of standard training and core competencies for Allergy & Immunology. In contrast, in 2019 the American Medical Association adopted policies to bolster incorporation of AI within medical education(71). Specifically, curricular modifications to incorporate educational modules and training on bias in assistive technologies were highlighted. As technological advances occur, medical education will increasingly be focused on ensuring trainees have mentors in informatics and senior faculty members who can contextualize AI data applications. Applying these technologies may become important for professional development and identity formation for future Allergy & Immunology specialists.

AI Adoption

Prior to adoption, AI algorithms should be rigorously validated in the context of clinical trials. To date only one comparative effectiveness study of an AI algorithm has been performed in the field of Allergy & Immunology(72). Clinical utility, algorithmic explainability and interoperability should also be considered. Safeguarding privacy of data and outlining liability concerns(73) are paramount. AI dataset governance and impacts on individual patients are similar to issues frequently discussed when considering genomic data(74). Data sharing amongst AI applications, the EHR and other health devices along with the impact of direct-to-consumer tools will raise further privacy and ethical concerns. Engaging policy makers and clinicians in advocacy efforts to establish privacy safeguards will be needed to preserve viable and safe AI systems. Rapid adoption may cause interoperability problems similar to rapid EHR adoption if downstream effects are not anticipated(75). Ultimately, the value of AI for specific applications in Allergy & Immunology will influence the degree and magnitude of AI adoption in our specialty.

Coverage and Reimbursement

In 2018 the Food and Drug Administration (FDA) granted approval to an AI device for retinal fundus examinations(76). This device called IDx-DR is capable of rapid autonomous retinal examination, can be used in areas of specialty shortages, and could be scalable and cost-effective(77,78). Tools such as IDx-DR fall under the FDA category of Breakthrough Device Designation (BDD) as novel technology, which have their own criteria and pathways to approval. However, Medicare Administrative Contractors (MACs) ultimately determine coverage of BDDs according to coding, coverage and payment rules related to medical necessity. A BDD may be eligible for New Technology Add-On Payment (NTAP), but this only applies to inpatient services. In contrast, outpatient device coverage will depend on commercial payers and provider contracts. As such, there is considerable ambiguity about how AI may be reimbursed in a comprehensive manner and whether these technologies will ultimately prove to be cost-effective. In addition, the impact of direct-to-consumer markets on the use and adoption of AI digital tools will influence cost for patients, clinicians and the healthcare system. Future areas of research must investigate the feasibility, efficacy and cost-effectiveness of AI implementation within mobile health devices, the EHR and in assessments of population health.

AI Maintenance and Evaluation

As clinicians find increasing use for AI tools, clinical acumen will need to be expanded toward interpreting AI. In addition, rigorous assessment of algorithmic explainability and reporting of clinical and performance measures, including the population in which the AI was trained and the standard to which the AI was compared using one of several approaches such as a checklist for manuscript writing such as MI-CLAIM, “Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis” (TRIPOD-AI) or “Prediction model Risk Of Bias Assessment Tool” (PROBAST-AI), or “Minimum Information for Medical AI Reporting” (MINIMAR) will be important, and are reviewed elsewhere(79). Added training will be needed to interpret appropriate construction of studies using the AI-CONSORT guidance(80), and as technology advances algorithms evaluated to assure they remain appropriate.

Future Directions

Al has the potential to impact the specialty of Allergy & Immunology in the next decade. From a data procurement perspective, advances in data systems, EHR design and recognition of relevant inputs (e.g. pollen identification(81,82), epitope maps) hold promise for AI-based improvements. As routinely generated medical and genetic data are captured and stored more thoughtfully within EHRs, more reliable data acquisition, and subsequent model development using evidence-based and best-practice algorithm training and validation will be possible. Use of AI for pathology and radiology relevant applications can be applied to improve care for patients with allergic and immunologic disorders(83-85). Additionally, post-market surveillance of outcomes and consideration of ethical factor impact on healthcare disparities will be required. Fundamentally, involvement and training of Allergy & Immunology physicians to be stewards in the design, development, and implementation of AI will be important to define best practice use for optimal patient outcomes.

Conclusions and Recommendations

This AAAAI workgroup report highlights the potential clinical implementation of AI in Allergy & Immunology to improve individual and population-wide health. This report serves as a high-level framework for conceptualizing AI for the practicing clinician. Limitations of this report include the inability to describe at length the process of data procurement, and model development that would help inform the practical considerations outlined throughout. Many aspects such as data quality, algorithm sharing and explainability, partnering with developers, facilitating data science competency in Allergy & Immunology training programs, use of AI in quality improvement, and validation of theoretical models are not discussed in sufficient depth for the experienced reader. Similarly, how and when to utilize AI or how to develop AI falls outside of the scope of this workgroup report. It should be noted that AI will not be the best fit for all data applications. In some cases, data scientists may determine that a rigorous statistical approach is more appropriate than an ML/AI approach, for example when a high degree of accuracy is desired, or for small datasets or when there are a limited numbers of variables(86). Furthermore, assuring equity dimensions and potential impact of AI on health disparities is key and we only scratched the surface in describing some potential impacts in this workgroup report. Future impact analyses of AI on clinician workflow, healthcare consults, and health disparities are needed. There are several concrete recommendations that can be made as AI continues to mature and evolve:

• Engage frontline clinicians and healthcare systems in developing AI systems that are cost-effective, abide by privacy and legal standards.

• Allergy & Immunology specialists should be involved in the design, validation and implementation of AI in Allergy & Immunology.

• Partnerships with various stakeholders including but not limited to local health system informaticists, clinicians, patients, data scientists, and health information technology professionals will be needed as AI in Allergy & Immunology is deployed.

• Payers and policymakers will have important inputs into applications of future AI.

• Clinicians engaging with AI should become trained in these domains, to optimally advance AI for Allergy & Immunology applications and patient care.

Abbreviations:

AAAAI

American Academy of Asthma Allergy and Immunology

AI

Augmented intelligence

AI-CONSORT

Artificial Intelligence Consolidated Standards of Reporting Trials

ANN

Artificial Neural Networks

BDD

Breakthrough Device Designation

CDS

Clinical Decision Support

COVID

Coronavirus Induced Disease

DL

Deep Learning

DNA

Deoxyribunucleic Acid

EGID

Eosinophilic Gastrointestinal Syndromes

eMERGE

Electronic Medical Records and Genomics

EoE

Eosinophilic Esophagitis

EP

Electronic Phenotyping

EHR

Electronic Health Records

FAERS

FDA Adverse Reporting System

FDA

Food and Drug Administration

GWAS

Genome Wide Association Study

HITE

Health Informatics Technology and Education Committee

IEI

Inborn Errors of Immunity

IgE

immunoglobulin E

IUIS

International Union of Immunological Societies

MACs

Medicare Administrative Contractors

MCDM

Multi-Criteria Decision-Making

MI-CLAIM

Minimum information about clinical artificial intelligence modeling

MIS-C

Multisystem Inflammatory Syndrome-COVID

ML

Machine Learning

mRNA

messenger ribonucleic acid

NLP

Natural Language Processing

NMR

nuclear magnetic resonance

NTAP

New Technology Add-On Payment

OMOP

Observational Medical Outcomes Partnership

PCN

penicillin

PID

Primary Immunodeficiency

rRNA

ribosomal RNA

SaMD

Software As a Medical Device

scRNAseq

Single cell Ribonucleic Acid Sequencing

SARS-CoV2

Severe Acute Respiratory Syndrome-Coronavirus 2

SNP

single nucleopeptide polymorphism

SVM

Support Vector Machine

T-REX

Tracking Responders Expanding

UNICEF

United Nations Children's Fund