Skip to main content
PMC home page
Journal of the American Medical Informatics Association : JAMIA logoLink to Journal of the American Medical Informatics Association : JAMIA
. 2021 Apr 25;28(7):1582–1590. doi: 10.1093/jamia/ocab065

Enhancing trust in AI through industry self-governance

Joachim Roski 1,✉, Ezekiel J Maier 1, Kevin Vigilante 1, Elizabeth A Kane 1, Michael E Matheny 2,3
PMCID: PMC8661431  PMID: 33895824

Abstract

Artificial intelligence (AI) is critical to harnessing value from exponentially growing health and healthcare data. Expectations are high for AI solutions to effectively address current health challenges. However, there have been prior periods of enthusiasm for AI followed by periods of disillusionment, reduced investments, and progress, known as “AI Winters.” We are now at risk of another AI Winter in health/healthcare due to increasing publicity of AI solutions that are not representing touted breakthroughs, and thereby decreasing trust of users in AI. In this article, we first highlight recently published literature on AI risks and mitigation strategies that would be relevant for groups considering designing, implementing, and promoting self-governance. We then describe a process for how a diverse group of stakeholders could develop and define standards for promoting trust, as well as AI risk-mitigating practices through greater industry self-governance. We also describe how adherence to such standards could be verified, specifically through certification/accreditation. Self-governance could be encouraged by governments to complement existing regulatory schema or legislative efforts to mitigate AI risks. Greater adoption of industry self-governance could fill a critical gap to construct a more comprehensive approach to the governance of AI solutions than US legislation/regulations currently encompass. In this more comprehensive approach, AI developers, AI users, and government/legislators all have critical roles to play to advance practices that maintain trust in AI and prevent another AI Winter.

Keywords: artificial intelligence/ethics, artificial intelligence/organization and administration, certification, accreditation, policy making

INTRODUCTION

Artificial intelligence (AI) has been touted as critical to harnessing value from exponentially growing health and healthcare data. AI can be used for information synthesis, clinical decision support, population health interventions, business analytics, patient self-care and engagement, research, and many other use cases. Clinician, patient, and investor expectations are high for AI technologies to effectively address contemporary health challenges.

However, prior periods of AI enthusiasm were followed by periods of disillusionment, known as “AI Winters,” where AI investment and adoption withered.1 We are now at risk of another AI Winter if current heightened expectations for AI solutions are not met by commensurate performance. Recent examples that highlight the growing concern over inappropriate and disappointing AI solutions include racial bias in algorithms supporting healthcare decision-making,2,3 unexpected poor performance in cancer diagnostic support,4 or inferior performance when deploying AI solutions in real-world environments.5 Such AI risks may be considered a “public risk,” denoting threats to human health or safety that are “centrally or mass-produced, broadly distributed, and largely outside the risk bearers’ direct understanding and control.”6 The public’s concerns about such risks that could contribute to a “techlash” or AI Winter have recently been documented.7

In a seminal report by the National Academy of Medicine (NAM), the authors detailed early evidence for promising AI solutions for use by patients, clinicians, administrators, public health officials, and researchers.1,8–12 In this article, we expand on that work by identifying 10 groups of widespread AI risks and 14 groups of recently identified mitigation strategies aligned to NAM’s AI implementation life cycle.

While AI governance efforts have been proposed previously,13,14 it remains unclear who (eg, government vs private sector/industry) is best positioned or likely to take specific actions to manage AI risks and ensure continued trust across a broad spectrum of AI solutions. The need for industry self-governance, which refers to the collective, voluntary actions of industry members, typically arises from broad societal concerns and public risks that governments may not be adequately addressing in their legislative or regulatory efforts.15 In this manuscript, we describe how AI risk mitigation practices could be promulgated through strengthened industry self-governance, specifically through certification and accreditation of AI development and implementation organizations. We also describe how such self-governance efforts could complement current government regulations and tort law to maintain trust in a broad spectrum of AI solutions for clinical, population health, research, healthcare management, patient self-management, and other applications.

AI risks and mitigation practices across the AI implementation life cycle

The recent NAM report on AI & Health described an AI implementation life cycle that can serve as an organizing schema to understand specific AI risks and mitigation practices. Figure 1 illustrates the 4-phase NAM AI implementation life cycle. Phase 1 defines clinical and operational requirements, documents the current state, and identifies critical gaps to be filled by AI development. Phase 2 encompasses the development and validation of AI algorithms for a specific use case and context. Phase 3 focuses on organizational AI implementation. Phase 4 focuses on continued maintenance and sustainment of implemented AI.

Figure 1.

Figure 1.

Open in a new tab

NAM AI/ML implementation life cycle.

Adapted and reproduced from: National Academy of Medicine. 2020. NAM Special Publication: Artificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril. Reproduced with permission from the National Academy of Sciences, Courtesy of the National Academies Press, Washington, DC.

We have summarized evidence for 10 groups of AI risks and 14 groups of associated evidence-based mitigation practices aligned to each phase of the NAM Life cycle in Table 1. While it is beyond the scope of this manuscript to provide an exhaustive summary of the relevant literature, Table 1 can serve as a convenient summary for stakeholders interested in translating evidence-based practices into future performance standards.

Table 1.

AI risks and mitigation practices across the AI implementation cycle

NAM Life cycle Risks Evidence-based practices
Phase 1: Needs Assessment

  • Lack of integration of stakeholder perspectives & considerations16–22

  • Lack of clearly defined organizational values & ethics23,24

  • User-centered design25,26

  • Organizational readiness assessment27–29

  • Organizational prioritization process1

  • User-centered workflow/change management process5,30–32
PHASE 2: Development

  • Data bias33–38

  • Lack of representative & equitable population33,39

  • Lack of data management37,40

  • No accounting for causal pathways41

  • Data transparency & reporting32,37,40,42–46

  • Model provenance records40

  • Promoting trust & explainability32,47–51

  • Distributed model development52
PHASE 3: Implementation

  • Lack of data encryption & privacy protections53,54

  • Lack of secure hardware

  • Lack of oversight for responsible AI adoption39,55

  • Equitable/diverse workforce

  • Organizational implementation38,46,47,56,57

  • Organizational governance13,47,58,59

  • Promote “human in the loop” practices60,61
PHASE 4: Maintenance

  • Lack of algorithmic accountability47,62

  • Performance surveillance33,63,64

  • Organization surveillance governance65
Open in a new tab

STRENGTHENING INDUSTRY SELF-GOVERNANCE TO PROMOTE TRUST-ENHANCING PRACTICES

Evidence-based AI risk mitigation practices should be more widely implemented by AI developers and implementers. Wider implementation could be ensured through government regulation of AI. However, such regulation is largely lacking in the US and elsewhere.66 Additionally, an initial group of AI developers, implementers, and other stakeholders could create new market expectations through collective, voluntary actions—industry self-governance—to identify, implement, and monitor adherence to risk mitigation practice standards.67

Industry self-governance can be contrasted with organizational self-governance. Organizational self-governance refers to the policies and governance processes that a single organization relies on to provide overall direction to its enterprise, guide executive actions, and establish expectations for accountability. Many prominent organizations have publicly declared their adoption of select, trust-enhancing AI risk mitigation practices that we described in the previous section. At the same time, there is divergence between these organizations about both what constitutes “ethical AI” and what should be considered best practices for its realization.68 Poor execution of organizational self-governance can result in damage to the institutional brand—and potentially open the organization to liability.69,70 It has been argued that a society’s exclusive reliance on organizational self-governance processes is unlikely to effectively ameliorate AI risks.71,72

Relying on industry self-governance in defining and monitoring adherence can offer several advantages. It has the potential to act faster and with greater technical expertise than government in defining and enforcing standards for products and services. It may also be more insulated from partisan politics, which can lead to legislative or regulatory deadlocks. Increased reliance on “regulatory oversight” through self-governance that is monitored by regulators has been proposed as a modernized approach to regulation in the age of rapidly evolving health technologies.73 Finally, in contrast to most government regulation, industry standards and enforcement mechanisms can reach across national jurisdictions to define and transparently enforce standards for products and services with global reach, such as AI.67

There is precedence for industry self-governance in the US healthcare sector. For example, a number of private sector healthcare accreditation and certification programs (eg, Joint Commission [JC] and National Committee for Quality Assurance [NCQA] accreditation, ISO9000 certification, Baldridge awards, etc) independently define and verify adherence to practice standards by hospitals, health plans, and other healthcare organizations, with accountability for patient safety and healthcare quality. In these efforts, private sector independent organizations, collaborate with healthcare industry organizations (eg, health plans or hospitals) and other experts to define relevant standards and performance metrics to improve healthcare safety and quality performance. These standards and metrics are based on research evidence, when available, or expert consensus when evidence is lacking or impractical to obtain. Additionally, these organizations also assess adherence to standards and measure performance through established, industry-vetted metrics. Due to the rigor and widespread use of these standards throughout the private-sector healthcare industry, government-run healthcare facilities (eg, Military Health Treatment facilities or Veterans Affairs Medical Centers) have adopted the same industry-defined standards and performance metrics. Similarly, the Centers for Medicaid and Medicare Services (CMS) condition payment/reimbursement of Medicare Advantage plans or healthcare facilities on the adherence to NCQA and JC standards and performance metrics. CMS’s deeming authority grants JC and NCQA the ability to demonstrate that their hospital and health plan clients meet or exceed CMS’s own standards for safety/quality. Once that has been demonstrated, JC or NCQA accreditation/certification is accepted by CMS in lieu of the agency inspecting these health organizations itself.

To counter growing mistrust of AI solutions,65,74 the AI/health industry could implement similar self-governance processes, including certification/accreditation programs targeting AI developers and implementers. Such programs could promote standards and verify adherence in a way that balances effective AI risk mitigation with the need to continuously foster innovation. Moreover, as described above in the instances of JC and NCQA, adherence to these standards could be equally expected of private and government-run AI developers and implementers.

PROMOTING AI RISK-MITIGATING PRACTICES THROUGH CERTIFICATION/ACCREDITATION

Based on other certification and accreditation programs referenced earlier, we next describe essential steps for the implementation of an AI industry self-governed certification or accreditation program. These steps are summarized in Figure 2 and explained in more detail below:

Figure 2.

Figure 2.

Open in a new tab

Steps to implement an accreditation/certification program.

  • Multistakeholder participation: Self-governance efforts requiring trust by a broad set of stakeholders must incorporate multiple perspectives. Stakeholders may include consumers/patients, clinicians and institutional providers, healthcare administrators, payors, AI developers, and relevant governmental agencies. Stakeholders could be effectively convened by an independent third-party organization (eg, a nonprofit organization) that has expertise in the field and enjoys the trust of all stakeholders. For example, the Consumer Technology Association has suggested potential standards for AI health solutions.75 A governing board of this organization should include representatives of all critical stakeholder groups in order to be credible and ensure that all perspectives are appropriately represented in a certification/accreditation program. Moreover, the organization’s governing board should also provide guidance to multiple committees for specific, detailed elements of the overall program (eg, standard development, performance metrics development, assessment/accreditation decisions, etc).

    Such an independent third-party organization could be a well-known, already established organization in a particular country, or an international organization with significant expertise that is able to operate in multiple jurisdictions. For example, the Institute for Electrical and Electronics Engineers (IEEE) has more than 417 000 members in over 160 countries and has long-standing experience in defining internationally adopted standards. It recently launched a Global Initiative on Ethics of Autonomous and Intelligent Systems and issued an iterative playbook of standards and best practices called, “Ethically Aligned Design,” which is intended to inform governments, organizations, businesses, and stakeholders around the world.14 To date, IEEE has not established a certification/accreditation program for AI developers and implementers. In addition, the World Economic Forum has also issued a model AI governance framework and assessment guide to be piloted around the globe.76

  • Develop consensus goals and framework: A stakeholder-consented framework to enhance trust in AI and certification/accreditation program goals must be developed to promote and verify effective implementation of risk-mitigation practices. Table 1 describes potential elements of such a framework that identifies AI risks and mitigation practices along an AI implementation life cycle. The formulation of an enduring framework and overarching program goals will allow for a careful and regular evolution of specific standards and assessment methods that is synchronized with the framework and program goals.

  • Operationalize program design: Accreditation typically ensures adherence to a wide range of diverse standards, whereas certification may refer to a smaller, narrower group of standards. For example, AI accreditation could refer to adherence to all standards of a comprehensive framework, whereas certification could be achieved for only a subset. In either case, several elements will require careful consideration by an accreditation/certification entity, including the following:

    • Determine the certifiable/accreditable entity. Clear definitions of the certifiable/accreditable entity must be identified. Should an organization, a specific program within the organization, or a product developed by the organization, be certified or accredited? Should both AI developers and implementers be certified/accredited and based on what group of standards? Moreover, the definition of the accreditable entity should be clearly operationalized and have reasonable stability over time. For example, defining the certifiable/accreditable entity at a product level may be challenging, as certain AI products may evolve in relatively short periods of time. Fundamental product change over short periods may run counter to rendering meaningful certifications/accreditation decisions, which typically are meant to be valid for much longer periods (eg, 2–3 years) and based on an assessment at a particular point in time.

    • Define standards. A range of standards should be defined in accordance with an overarching framework and program goals. In Figure 1, we have identified a framework which aligns evidence for groups of standards for each phase of the AI implementation life cycle. Within each phase, individual standard groups can be identified based on evidence that makes up the “group” of standards for that phase. When defining standards, it is also important to define specific elements that an assessor must verify to determine if that standard has been met. It is plausible that different sets of standards might apply to AI development organizations and AI-implementing organizations, respectively, based on their different range of activities along the AI implementation life cycle. Organizations that both develop and implement AI solutions (eg, a large health system with resources and know-know to both develop and implement AI solutions) might be subject to a combined set of standards.

    • Measure adherence to standards and practices. A measurement system must be developed that allows for an independent verification of whether entities have met the standards. For instance, it must be determined what “evidence” is required to measure how a standard has been met (eg, review of submitted documents, calculation of submitted performance measures, onsite observation, etc). Additionally, processes must be implemented to ensure measurement methods are (1) valid (eg, assessment accurately verifies adherence to a standard/practice); (2) reliable (eg, different reviewers reach the same result); and (3) the least burdensome.

    • Establish periodicity for recertification or accreditation. AI organizations, programs, methods, and products advance rapidly. A viable certification/accreditation program must measure adherence to standards of a rapidly evolving industry. It also must strike the right balance between ensuring meaningful adherence standards without stifling ongoing innovation and improvements over time.

    • Continuously review standards and methods. Standards and assessment methods should be dynamic and adapt to evolving practices. Additionally, certification/accreditation programs may become more stringent and rigorous over time as experience increases with standards, assessment methods, and shifting practices.

  • Create market demand: The likelihood of effective industry self-governance depends on several factors. This includes, but is not limited to, the extent to which demand for firms’ products or services relies on their brand quality or the probability of collective action by stakeholders to exert pressure on an industry to address perceived risk.15 Verified adherence to best practices through certification/accreditation can improve AI developers’ and implementers’ brand through the ability to publicize adherence to a “good housekeeping” seal of approval. For example, being branded as a trusted developer and user of AI products or services may increase demand from customers, including hospitals, health systems, health plans, physician practices, and individuals. A similar approach helped establish health plan accreditation in the mid-1990s, when some large employers began demanding that health plan products they intended to purchase on behalf of their employees meet the criteria or standards for best practices established by NCQA.77

     The public sector (ie, federal, state, and local entities), in their roles as either payors or regulators, can similarly promote market demand by giving preferential treatment to AI developers and implementers adhering to private sector defined and implemented accreditation/certification programs. To accomplish this, US government agencies could exercise deeming authority by recognizing private sector certification/accreditation programs that ensure adherence to AI best practices, in lieu of submitting their products or services separately to a public sector review. For example, US hospitals accredited by a private-sector organization, such as the JC, can elect to be “deemed” as meeting CMS requirements by submitting to the review process of that private sector accrediting entity. The public sector can also gradually increase the expectations of what private sector accrediting organizations must address to be deemed.78

  • Evaluation of program effectiveness. Finally, certification/accreditation programs should be evaluated to ensure they meet their objective of increasing trust and adherence to best practices. Such evaluation can help determine if the program continues to meet critical private and public sector policy goals for more responsible AI development and implementation. If it is determined that the certification/accreditation program is not effective in managing AI risks, industry or government can decide to strengthen the program or market conditions that would make the program more effective.

INDUSTRY SELF-GOVERNANCE, REGULATION, AND LIABILITY

To date, the rise of AI has largely occurred in a regulatory and legislative vacuum. Apart from a few US states’ legislation regarding autonomous vehicles and drones, few laws or regulations exist that specifically address the unique challenges raised by AI.66

Industries across the globe have at times defined, adopted, and verified their adherence (eg, certification/accreditation) to beneficial standards in lieu of or as a complement to government regulation. When effective, industry efforts of defining, adopting, and verifying adherence to needed standards, can reduce the urgency of regulation through the public sector and afford the opportunity to invest limited public resources otherwise.14 Industry self-governance has the additional advantage of being able to establish standards for globally distributed products and services across jurisdictions, reducing the potential of inconsistent regulations, as well as the need and resources potentially required to achieve international harmonization of government regulations at a later point.

If industry self-governance is lacking or relevant legislation or political will already exists, and resources are available, government agencies can reserve the right to institute their own AI programs. One example of a government-implemented program that incorporates several of the aforementioned elements is the US Food and Drug Administration’s (FDA) software as a medical device (SaMD) certification program.13 In this voluntary program, SaMD developers who rely on AI in their software are assessed and certified by demonstrating an organizational culture of quality and excellence and a commitment to ongoing monitoring of software performance in practice.79–82 However, AI-enabled SaMD represents only a small portion of AI solutions deployed in health and healthcare. Others have suggested that additional legislation or efforts may be needed to manage AI risks across a broader range of AI health solutions. For example, it has been suggested that an Artificial Intelligence Development Act (AIDA) is needed to task an organization or government agency with certifying the safety of a broad range of AI products/systems across industry sectors.66

Approaches towards establishing greater accountability for AI developers and implementers through industry self-governance programs or regulation do not obviate the need for addressing legal liability. Unlike an accrediting organization or regulatory agency which would typically become active before harm from AI products occurs, courts are reactive institutions as they apply tort law and adjudicate liability in individual cases of alleged harm. To date, courts have not developed standards to specifically address who should be held legally responsible if an AI technology causes harm.66 Consequently, established legal theory would likely hold providers who rely on AI liable for malpractice in individual cases if it is proven that they owed (1) a professional duty to a patient; (2) that they were in breach of such duty; (3) that that breach caused an injury; and (4) that there were resulting damages.83 In order to establish legal links between certification and liability, AIDA could stipulate a certification scheme under which designers, manufacturers, sellers, and implementers of certified AI programs would be subject to limited tort liability, while uncertified programs that are offered for commercial sale or use would be subject to stricter joint and severable liability.66 A more in-depth exploration of legal liability is beyond the scope of this article, but both liability and self-governance can promote greater accountability for ameliorating AI risks.

CRITICAL CONSIDERATIONS FOR EFFECTIVE SELF-GOVERNANCE

There are a number of critical success-factors, as well as risks, or potentially unintended consequences that need to be considered and mitigated when relying on industry self-governance as a complement to other legislative or regulatory efforts to foster responsible use of AI.

In the US, the FDA is, as described earlier, currently offering certification for AI solutions, such as medical devices.13 However, the FDA’s current authority does not extend to most types of AI solutions supporting health/healthcare needs such as population health management, patient/consumer self-management, research/development, healthcare operations, etc. At the same time, some of the most prominent failures of AI solutions to deliver on their promise, therefore jeopardizing trust, pertain to AI solutions not covered by the FDA.2–5 This large segment of highly visible AI solutions in health/healthcare may be an appropriate focus for self-governance efforts to maintain trust.

While self-governance efforts in health/healthcare have proven to be successful in complementing legislative or regulatory efforts, several risks to effective self-governance should be managed carefully. Generally speaking, self-governance will fall short when the costs of self-governance to industry are higher than the alternatives. For example, success of self-governance may be less likely if the following conditions aren’t present or are not being created: a) the public sector signaling pending legislative actions to establish greater accountability for AI health solutions (eg, through expanded regulatory authority), and that government would accept self-governance programs in lieu of implementing its own programs to ensure accountability; b) perceived public pressure (eg, through public media) on industry to create more trustworthy products; c) private and public sector commitment to preferentially purchase AI solutions that have been certified/accredited; and d) a prominent initial (small) set of organizations (AI developers/users) willing to collaborate under the auspices of an independent organization to define standards and hold themselves accountable to them, thereby creating a market expectation for certification/accreditation for AI health solution developers or implementers. Since many private companies, research institutions, and public sector organizations have issued principles and guidelines for ethical AI, there may be a significant number of organizations interested in initiating such self-governance efforts.68

Importantly, self-governance is likely only successful if all stakeholders have confidence that standards and verification methods were developed by appropriately balancing perspectives of consumers/patients, clinicians, AI developers, AI users, and others. To that end, as described earlier, it is imperative that a third party, independent organization (eg, rather than a trade organization representing 1 stakeholder group), is charged with the development of standards and verification methods. Balanced development/oversight processes, resulting in meaningful and operationally “achievable” performance standards, avoid the risk of standards/verification methods being perceived as self-serving for industry. However, standards need to be created that don’t stifle innovation by being unnecessarily restrictive or by creating “high-costs” for accreditation/certification that may deter some AI developers from continuing to develop valuable AI health solutions.

To initiate the self-governance processes through an independent organization, start-up funding by the public sector or private-sector foundations or a group of organizations may be necessary. Such funding could support the independent organization in convening stakeholders and defining an initial set of standards and verification methods. Ongoing maintenance of standards and certification/accreditation program operations would likely need to be funded by fees levied on those organizations seeking certification/accreditation. Such a model is analog to the funding/business models of other health/healthcare certification/accreditation efforts.

CONCLUSION

The advancement of AI is actively being promoted by the US government,84–86 governments and policy makers of other countries,87 and supranational entities (eg, the European Union).88 However, signs of a “techlash” and the acknowledgment of disconcerting AI-related risks and challenges are also abundant.

Governmental management of public risks such as AI risks typically occurs in democratic societies through actions of the legislative, executive, and judicial branches of government. However, as described, AI-specific legislation, regulation, or established legal standards or case law largely do not exist worldwide—or they apply only to a narrow subset of AI health solutions. At the same time, many countries are hesitant to create national industrial policy approaches that may risk disadvantaging its industries during an intense global “competition” as the Fourth Industrial Revolution unfolds, dominated by smart technologies, AI, and digitalization.89 In 2020, the US government issued a report on AI that directed federal agencies to avoid regulatory or nonregulatory actions that needlessly hamper AI innovation and growth. The report identified ensuring trust in AI as the #1 principle of stewardship of AI while encouraging reliance on voluntary frameworks and consensus standards.90

The AI and healthcare industry could step in to manage AI risks through greater self-governance.14 We presented a framework to increase trust in AI that maps known AI risks and their associated, mitigating, evidence-based practices to each phase of the AI implementation life cycle. We also described how this framework could inform the standard development for certification/accreditation programs for a broad spectrum of AI health solutions that is not covered through current regulation.

Potential future legislation and regulation across the globe will, in the coming years, likely differ in terms of managing specific AI risks. However, encouraging the use of evidence-based risk mitigation practices, promulgated through self-governance and certification and accreditation programs, could be effective and efficient across national jurisdictions in promoting and sustaining user trust in AI, while staving off another AI Winter.

FUNDING

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

AUTHOR CONTRIBUTIONS

JR and MEM developed the concept and designed the manuscript; KV and EJM provided key intellectual support, and EAK provided research support and helped edit the manuscript.

DATA AVAILABILITY STATEMENT

There are no new data associated with this article. No new data were generated or analyzed in support of this research.

CONFLICT OF INTEREST STATEMENT

None declared.

REFERENCES

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

There are no new data associated with this article. No new data were generated or analyzed in support of this research.

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of Oxford University Press

RESOURCES