Rules for robots, and why medical AI breaks them

Abstract

This article critiques the quest to state general rules to protect human rights against AI/ML computational tools. The White House Blueprint for an AI Bill of Rights was a recent attempt that fails in ways this article explores. There are limits to how far ethicolegal analysis can go in abstracting AI/ML tools, as a category, from the specific contexts where AI tools are deployed. Health technology offers a good example of this principle. The salient dilemma with AI/ML medical software is that privacy policy has the potential to undermine distributional justice, forcing a choice between two competing visions of privacy protection. The first, stressing individual consent, won favor among bioethicists, information privacy theorists, and policymakers after 1970 but displays an ominous potential to bias AI training data in ways that promote health care inequities. The alternative, an older duty-based approach from medical privacy law aligns with a broader critique of how late-20th-century American law and ethics endorsed atomistic autonomy as the highest moral good, neglecting principles of caring, social interdependency, justice, and equity. Disregarding the context of such choices can produce suboptimal policies when - as in medicine and many other contexts - the use of personal data has high social value.

I. INTRODUCTION

The White House Office of Science and Technology Policy published a Blueprint for an AI Bill of Rights in October 2022.¹ The quest to codify rights of humans as against smart machinery dates at least as far back as Isaac Asimov’s three laws for robots in 1942.² It feels more pressing as smart machines crawl out of science fiction and into everyday life driving cars, serving ads for products an algorithm inferred you might want, screening job applicants, nixing people’s eligibility for home loans and parole, diagnosing whether a skin lesion is cancer, and advising the doctor how to treat it if it is.³ Carlos Ignacio Gutierrez and Gary Marchant recently counted 634 soft law programs worldwide striving to enunciate rules to protect the interests of humans in the age of AI.⁴

The White House’s Bill of Rights analogy presumes that a single, overarching set of norms can protect people’s interests in all contexts where artificial intelligence/machine learning (AI/ML) tools are used. This article questions that view, using an example drawn from medical AI as a proxy for a larger phenomenon, which is that many uses of AI tools raise context-specific legal and ethical concerns for which general rules are poorly tailored. Inspired by Helen Nissenbaum’s influential insight that privacy norms are contextual,⁵ this article suggests more broadly that all rules for robots and AI systems—not just the privacy rules—may need to be contextually informed and context-appropriate.

The love of general rules may be a uniquely human failing and, if so, it may prove to be the tragic flaw that dooms the struggle to protect human rights against smart robots and AI systems. Complexity and diversity are hard for humans. The human mind can only balance several factors—perhaps two to five—simultaneously. There is an inherent ‘mismatch between the mathematical optimization in high-dimensionality characteristic of machine learning and the [more limited] demands of human-scale reasoning.⁶’ Machine brains can weigh thousands of factors at once and cope with diversity and nuance unemotionally. If nuanced, context-appropriate rules are what humans need to survive against smart machines, then humans may lose, because humans are saddled with low-dimensionality, two- to five-factoral minds apt to compress complex realities into simplistic lists of general, all-encompassing rules, as the White House Blueprint for an AI Bill of Rights just did.

AI/ML clinical decision support (CDS) tools—the example this article explores—are a category of medical software providing ‘knowledge and person-specific information, intelligently filtered or presented at appropriate times, to enhance health and health care.⁷’ CDS tools combine information about an individual patient with various sources of general medical knowledge to develop patient-specific diagnostic or treatment recommendations.⁸ This general medical knowledge can include insights from peer-reviewed literature, practice guidelines, and FDA-approved drug labeling, but in the case of AI/ML CDS tools, it often include inferences generated internally by an AI/ML algorithm processing clinical health data drawn from many other patients treated in the past.⁹

AI/ML CDS tools are designed to assist (not replace) health care professionals in traditional health care settings such as clinics, hospitals, and nursing homes.¹⁰ They represent a large, important category of medical AI, rather like a flagship in the fleet now sailing into the future of AI-enabled clinical health care. The clinical use context distinguishes AI/ML CDS tools from consumer-facing health apps and health-inflected AI tools intended for use by laypeople,¹¹ and from AI algorithms imbedded in traditional medical devices (such as software processing mammograms to highlight areas suspicious for disease).¹² AI/ML CDS tools infer what might be wrong with a patient and how best to treat it by comparing information about that patient to lessons gleaned from many other people’s prior health care experiences. ¹³ They thirst for data (more data, better-quality data, more diverse and inclusive data reflecting all of us) and raise privacy concerns. They are already part of the workflow at many health care facilities, recommending diagnoses and treatments to health professionals and influencing patients’ care, often without the patients knowing AI is running in the background, suctioning their data,¹⁴ and guiding important decisions about their health.¹⁵

The contexts where AI tools are employed present distinct challenges and opportunities. These can overwhelm attempts to state one general set of rules for AI used in large-scale data collection and processing in modern surveillance societies.¹⁶ Part I surveys rules proposed for robotic and AI systems generally and argues that AI-enabled health care may require different rules, including different privacy rules, to protect individual rights while promoting social justice.

In making this argument, this article is aware that calls for medical exceptionalism often prove spurious. Policymakers still struggle to move past a recent error of this sort, genetic exceptionalism, which was the belief that ‘genetic tests should be treated differently from other laboratory tests for oversight purposes.’¹⁷ It sprang from deliberations of a federal task force working in 1991, just as the Human Genome Project was starting.¹⁸ Scholars feared the genome ‘holds dark, implacable secrets’ and ‘toxic knowledge’ that might overwhelm genetically unschooled users and require special regulatory guardrails.¹⁹ As scientists learned more about the uses (and the limits) of genetic tests, the case for exceptionalism grew weaker.²⁰ Genetic tests, it turned out, were not all that different from many other diagnostic tests used in clinical and other settings. Medical AI may simply be the latest new thing that ‘holds dark, implacable secrets’ and ‘toxic knowledge’ in the minds of scholars hoping for something exceptional to write about.

With genetic exceptionalism as a cautionary example, proponents of exceptionalism should state what, precisely, warrants special treatment. Part I tries to do that for medical AI. If medical AI does, in fact, warrant special rules, this creates a jurisdictional quandary. Pervasive digital surveillance has blurred the line between medical and non-medical software. AI tools operating outside clinical health care settings can spot patterns in non-medical data and draw inferences about a person’s health. ²¹ If medical AI deserves special rules, should they apply to all AI that produces or processes health-related data, or just to AI intended for use in clinical health care? Part II argues that rules for AI should be tailored to the use context, rather than the data content, and suggests an intent-based approach for distinguishing which software must answer to special rules.

Any context-specific analysis—like this one—is only as generalizable as the context for which it was designed. Medical AI serves in this article as a metaphor, representative or symbolic of something else: that there may be limits to how far legal and ethical analysis can abstract AI, as a category, from the myriad use contexts in which AI is deployed in modern societies. General rule sets—‘carefully-delineated statements as to the thin common denominator of the wide legal terrain covered by the wholesale category’ of AI—can be enormously clarifying as long as we stay mindful that ‘the correct level of abstraction is a function of the purpose of our inquiry’.²² A crucial step when assessing social impacts of AI is to distinguish whether the problem is the AI or other, preexisting conditions of the environments where AI is in use.

II. DIFFERENT RULES FOR DIFFERENT ROBOTS

To identify candidate rules for AI tools, this discussion takes as its signposts the ‘basic ethical principles’ set out in the 1979 Belmont Report, a foundational work in American medical research ethics.²³ It supplements these principles with three influential statements of ‘rules for robots’ from legal and popular literature: those of Isaac Asimov, Jack Balkin, and Frank Pasquale.²⁴ Their compact rule sets brim with insights and express shared truths and fears about human relations with smart machines and algorithms. After surveying the rules, this Part explores whether rules proposed for robots in general are well tailored for the special case of AI/ML CDS tools in clinical health care. Several of the rules do seem relevant. More troubling are a couple of the rules with question marks hanging over whether CDS tools comply with them or, indeed, whether they are even the right rules for medical software at all.

II.A. From Asimov’s Rules for Robots to the Belmont Report: The Rise of Autonomy and Consent Norms

Isaac Asimov’s three laws for robots state broad ethical principles: (i) ‘a robot may not injure a human being or, through inaction, allow a human being to come to harm’; (ii) ‘a robot must obey orders given to it by human beings except where such orders would conflict with the First Law’; (iii) ‘a robot must protect its own existence as long as such protection does not conflict with the First or Second Laws’.²⁵

Asimov’s laws arose in the same climate of mid-20th-century postwar liberalism—with its commitments to individual liberty, human rights, consent, and fairness—that also fueled the American bioethics movement in the late 1950s through 1970s.²⁶ It is thus not surprising that Asimov’s laws resemble the ‘basic ethical principles’ of the Belmont Report, which inspired over 40 years of subsequent bioethical thought about ethical treatment of people volunteering for biomedical and behavioral research.²⁷ The Belmont principles are ‘respect for persons’, ‘beneficence’ (which encompasses its do-no-harm twin, nonmaleficence), and ‘justice’.²⁸

Like the Belmont principles, Asimov’s First and Second Laws emphasize robotic respect for persons and beneficence/nonmaleficence toward human beings. Asimov’s Second Law requires robots to obey human orders: that is, to act with human consent. His Third Law elevates these principles of respect for humans, beneficence/non-maleficence toward them, and human consent above other considerations, including the robot’s own survival. Justice has a minor, supporting role if robots are forced under Clause 2 of Asimov’s Second Law to adjudicate whether a human’s command contravenes the First Law against injuring human beings.

A similar skew emerged in American bioethics after the Belmont Report. Several senior scholars in the field liken bioethics to ‘a lopsided table with four legs’ in which ‘autonomy trumps other contenders’—that is, the other Belmont principles of beneficence, nonmaleficence, and justice.²⁹ This skew flows from a choice about what ‘respect for persons’ entails. Post-Belmont bioethics embraced an atomistic notion of personhood, based on an ‘assumption of human separateness, where freedom is fundamental’ and ‘autonomy exercises radical self-direction’.³⁰ If personhood is viewed this way, then ‘respect for someone as a person, as a chooser’ requires a strong norm of informed consent.³¹ After the Belmont Report, American bioethics embraced this view, consecrating individual consent as its emblem of respect for persons, ‘elevat[ing] the principles of autonomy and self-determination above other competing values in the hierarchy of ethical goods, such as beneficence, justice, dignity, and equality’.³²

This choice was, in many ways, historically contingent. ‘The cultural preoccupation with individual autonomy is a distinctly post-World War II social phenomenon’ and ‘[c]ontemporary American bioethics developed in that milieu’.³³ It was not, however, the only choice ethicists could have made. An alternative, social (relational) concept of personhood portrays individuals as fundamentally social, deriving identity from professional and personal relationships and ‘an array of competing communal principles that both identify the self and empower it to act’.³⁴ This is the view, recently gaining ground in popular American culture, that identity rests not just on rugged independence but also on each person’s connections within a web of class, gender, racial, socioeconomic, and other affiliations. Proponents of this social-self school of thought, had they exerted greater influence on post-Belmont bioethics, might have set it on a path stressing other ethical principles: beneficence, justice, and health equity.

Instead, as things stand, the focus on consent subordinates the principle of beneficence to a degree that makes it hard, four decades later, to recall what ‘beneficence’ meant at the time it became the second Belmont principle.³⁵ The principle of beneficence was not a mere platitude exhorting people to be kind but referred to an ‘an older ethic of physician beneficence’ that stressed physicians’ duties to ‘act as entrusted fiduciaries’ in the context of the social relationship between the physician and patient.³⁶ Justice also takes a backseat to autonomy and consent: ‘The major moral and conceptual problems about informed consent are not justice-based and do not directly confront issues of social justice’.³⁷

Exposed to decades of this brand of bioethical thinking, many Americans cited autonomy in refusing to comply with public health measures to protect others during the recent pandemic, and many feel ‘privacy is something to be protected at the discretion of the individual to whom the information relates’.³⁸ Daniel Solove paraphrases modern ‘control-over-information’ privacy theory as ‘we protect all information over which individuals want to retain control’.³⁹ This theory calls for individuals to consent before secondary uses of their identifiable health information, even if other people might have valid moral claims to the same information and even if the secondary uses would advance the wellbeing of other people, society in general, and the person the data describe.

The belief that data use should require consent is so widely held today that younger readers may be unaware of how recently it arose. The Belmont report focused mainly on clinical research but sparked a debate about whether its principles should also apply to informational research that uses people’s health data.⁴⁰ A Privacy Protection Study Commission (PPSC) created by the Privacy Act of 1974 noticed that, as of 1977, medical researchers were routinely using patients’ clinical health records in research without consent.⁴¹ The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research confirmed that, as of 1978, consent was not the norm in informational studies ‘based exclusively upon existing records, data or materials [e.g., biospecimens] gathered for other purposes’.⁴² The PPSC recommended that people should be asked for consent before researchers use their identifiable health information.⁴³

The postwar generation of bioethicists coming of age as scholars in the late 1970s embraced this norm of informed consent for research uses of personal data.⁴⁴ Equivalent notice-and-consent norms later gained favor in the ‘Information Privacy Law Project’ examining data privacy in broader (non-medical) social and economic contexts.⁴⁵ The concept of ‘privacy as a personal right to control the use of one’s data’ enjoyed ‘staggering’ consensus 20 years ago and was the ‘leading paradigm on the Internet and in the real, or off-line world’.⁴⁶

Even 20 years ago, there were critics of control-over-information theory, which equates privacy with individual consent rights. Anita Allen noted that people who consent to share deeply personal information exercise control over the information, but the act of consenting entails losing a bit of what most people commonly view as privacy.⁴⁷ More recently, it emerged that individuals’ privacy is interdependent. Others who consent to share information about themselves may inadvertently reveal facts about you, as when another partygoer’s selfie displays you passed out drunk in the background.⁴⁸ Initially, this interdependency drew scholarly interest only in specific contexts, for example, when genetic testing reveals facts about family members⁴⁹ or when research involving Indigenous people allows stigmatizing statistical inferences about all members of small tribal communities.⁵⁰ In truth, the problem is more general. Privacy interdependency potentially affects everybody in a world of large-scale, generalizable data analytics. If people who are representative of you consent to be studied, a study can reveal facts about you even when you opt out.⁵¹ Privacy is not simply a function of who has access to your input data, which was how control-over-information theory conceived it. Privacy also depends on what can be inferred about you. You ultimately cannot control what other people infer.

In modern information economies, privacy loss is in part systemic, an inevitable consequence of the kinds of data analytics modern societies choose to pursue. The individual’s choice to contribute or withhold personal input data from those analyses will not necessarily prevent unwanted personal inferences from being drawn. Just as de-identification gradually loses its power to protect privacy as algorithms grow adept at inferring the stripped-off identifiers,⁵² so consent loses its power to protect privacy as algorithms grow less biased, more generalizable, better able to draw inferences about the non-consenters who opt out.

Despite these doubts about consent as a privacy protection, consent rights remain popular with the public and with many bioethicists and policymakers, who may pragmatically view consent rules as simpler and less costly than designing computer systems that would shield individuals from unwanted revelation of personal facts (whether by disclosure or by inference). The White House’s Blueprint for an AI Bill of Rights shrugs off modern concerns that consent is unable to bear the weight of protecting privacy. It recommends a notice-and-consent privacy scheme in which ‘designers, developers, and deployers of automated systems’ must ‘seek your permission’ to use data in an AI system.⁵³ This aligns the Blueprint with the control-over-information theory that emerged after the Belmont report. Control-over-information theory is discredited, but it is far from dead.

The Belmont principles map Isaac Asimov’s rules for robots onto a set of rules for the humans who design and operate the robots and AI/ML software. The mapping is strikingly close for data privacy norms. Asimov’s rules require a robot to respect and protect humans, act with their consent, and destroy itself if it cannot do both. The Belmont principles call on biomedical researchers to respect persons who contribute data and biospecimens to research and to act with their consent. All that seems missing is an imperative for biomedical researchers to sacrifice their research mission, if necessary, to uphold consent norms. This, in fact, is not missing and is implicit in statements like this one by Franklin Miller of the National Institutes of Health’s Department of Bioethics:

The facts that historically much valuable population-based observational [informational] research has been conducted without informed consent, that obtaining consent would often make such research impossible to conduct, and that selection biases associated with soliciting consent may compromise its scientific validity do not in themselves constitute valid ethical reasons for waiving a requirement of informed consent.⁵⁴

This statement applies Asimov’s Third Law to the facts of informational research. The ‘robot’ (in this analogy, informational research that has high social value) must obey Asimov’s Second Law (to follow orders/act with consent), even if doing so destroys the ‘robot’ itself—that is, even if requiring consent undermines the scientific validity of results. By this view, the moral imperative to obtain consent is so strong that science must sacrifice its own validity, if necessary, to uphold the norm of informed consent.

II.B. The Ethic of Beneficence in a Control-over-Information World

Like the Belmont report, Jack Balkin and Frank Pasquale aim their rules for robots not at the robots, but at the humans who program them, operate them, use them, and ‘allow themselves to be governed by them’.⁵⁵ Their rules address broader duties of users and controllers of AI systems, going well beyond privacy and the ethics of data use.⁵⁶ This section discusses Prof. Balkin’s first rule, which relates to privacy and data ethics and defers their other rules to the next section.⁵⁷

Prof. Balkin’s three laws would, first, require humans who are responsible for AI/ML algorithms to act as ‘information fiduciaries’ of their clients, customers, and end-users to whose data they have access.⁵⁸ Second, responsible parties would have ‘public duties’ to uninvolved third parties—ie affected persons with whom they are not in privity of contract.⁵⁹ Third, the responsible parties would have a duty not to externalize costs and harms of their operations.⁶⁰

Absent from Prof. Balkin’s rules are the consent norms that figure so prominently in Asimov’s rules and in post-Belmont data ethics. In other work, he challenges scholars to ‘shift the focus’ in discussions of privacy ‘from the kind of information to the kinds of relationships—relationships of trust and confidence—that governments may regulate in the interests of privacy’.⁶¹ His ‘central point is that certain kinds of information constitute matters of private concern not because of their content, but because of the social relationships that produce them’.⁶² His first rule de-centers consent as a privacy protection and instead calls for placing those who handle people’s personal information under strong duties to be careful with it.⁶³

This approach aligns with the relational view of personhood that, according to Alfred Tauber, was the road not taken by American bioethicists in the early years after the Belmont Report.⁶⁴ Prof. Balkin’s first rule reinstates the older ‘ethic of beneficence’/‘ethic of care’—the second Belmont principle—that imposed a ‘primary ethical obligation’ for data handlers ‘to be responsible’ for protecting the privacy of patients whose data they receive.⁶⁵

After the Belmont Report, many bioethicists and information privacy scholars converged on control-over-information theory, which ‘conceives of privacy as a personal right to control the use of one’s data’ and stresses individual consent rights as the way to protect privacy.⁶⁶ This convergence, however, was never complete. Medical privacy law—the law governing privacy in hospitals, clinics, and other clinical health care settings—stubbornly rejects control-over-information theory.⁶⁷ When you enter a clinical health care setting, the legal reality is that you will have very limited control over your information.⁶⁸ Most medical privacy laws, including the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule,⁶⁹ allow secondary use of data if the patient authorizes (consents to) the use or if the data have been de-identified, but those are just two among many other data-sharing pathways.⁷⁰ The HIPAA Privacy Rule, for example, creates 27 legal pathways for secondary use or sharing of people’s health data, of which 22 require no consent.⁷¹

This situation is not a uniquely American ‘Wild West’ aberration. The European Union’s General Data Protection Regulation (‘GDPR’),⁷² which many Americans tout as a paragon of strong consent norms, grants the EU Member States leeway to establish their own medical privacy laws.⁷³ The Member States’ laws enable many of the same unconsented data flows that the HIPAA Privacy Rule allows.⁷⁴ Data flows, including some unconsented ones, prove unavoidable in order for health care systems to function and serve their essential contextual values—for example, treating the sick, tracking and managing epidemics, detecting domestic violence, facilitating organ transplants, and enforcing health care quality standards.⁷⁵

Medical privacy law ‘regards autonomy as only one of a number of moral principles governing the doctor-patient relationship’ and requires a ‘better balance of patient rights and physician responsibilities’.⁷⁶ Unlike most modern information privacy laws, medical privacy law does not focus on restricting the ‘upstream’ collection of data into the clinical health care system; instead, it follows a ‘downstream data protection model (“confidentiality”)’ that limits disclosure of clinical information after the health care system has already collected or generated it.⁷⁷ This orientation reflects the reality that broad collection of patient data is a necessary part of providing health care, and patients will more willingly share needed data with their providers if the latter are bound by confidentiality duties restricting downstream data redisclosure. As already noted, medical privacy laws do allow downstream redisclosure of clinical data for a variety of purposes without consent.⁷⁸ The allowed disclosures are for a narrow list of enumerated purposes that have high social importance and take place in a context where fiduciary duties and professional norms discourage overdisclosure.⁷⁹

The diminished role consent plays in medical privacy law may in part reflect skepticism about whether an ‘upstream’ consent model (seeking consent at the point of data collection) can protect data privacy in clinical settings.⁸⁰ When the PPSC recommended, late in the 1970s, that use of health data should require individual consent, this recommendation was for data used in biomedical research, not for clinical health care.⁸¹ Free, uncoerced consent can be a meaningful concept in research settings as traditionally conceived, where people weighed whether to contribute their data to a research study that, by definition, had no therapeutic aims.⁸² In contrast, people enter the clinical health care setting ill, in pain, and needing care, sometimes urgently. They seemingly would agree to any data use furthering the health care they are seeking, and their desperation undercuts consent as a tool for protecting data privacy in clinical contexts.

In clinical health care, the longstanding approach for protecting privacy—dating not just to the early 1970s but 2400 years back to Hippocrates—requires health care providers to ‘act as entrusted fiduciaries’ for patients, who are considered too vulnerable to meaningfully self-protect their privacy by refusing to share data with their doctor.⁸³ Medical privacy law never embraced the late-20th-century optimism that ‘autonomy as a construct’ can ‘replace the ethical responsibilities of the caregiver’.⁸⁴ Is it better to have people consent to do business with privacy violators, or to impose duties for data handlers not to violate people’s privacy? Medical privacy law took the latter approach, although there are grounds to debate whether the duties it imposes are adequate.⁸⁵ Consent remains hallowed in medical circles as a way to demonstrate respect for persons, but without the illusion that consent is a two-for-the-price-of-one proposition that simultaneously protects their privacy.

The United States relies on a web of state and federal laws defining responsibilities of clinical data handlers. State medical records laws govern the collection, use, and retention of data from medical treatment encounters and set procedures for sharing the records and for their disposal or transfer if the physician/patient relationship ends.⁸⁶ State courts enforce common-law duties for health care providers (eg, physicians, nurses, clinics, hospitals) to protect confidential information they hold, and most states supplement their common law with statutes addressing the privacy of various kinds of data held by health professionals and facilities.⁸⁷ Providers’ duties of confidentiality extend not just to their patients’ data but to information about other people, such as a patient’s family members, that sometimes makes its way into a patient’s files.⁸⁸

General health laws that seem on their face to be unrelated to privacy incorporate significant privacy protections. For example, hospitals breaching patient confidentiality would violate hospital accreditation standards, state facility license terms, and conditions of participation in the Medicare program.⁸⁹ The resulting sanctions could put a loose-lipped hospital out of business. State professional licenses for physicians and nurses focus mainly on professional skills and competence but also address patient confidentiality; violations can lead to loss of licensure.⁹⁰ Nongovernmental bodies such as the American Medical Association add an additional layer of soft law confidentiality norms.⁹¹ At the federal level, the HIPAA Privacy Rule preempts weaker state medical privacy laws while letting more stringent state provisions continue in effect.⁹² Other federal regulations, commonly called the ‘Part 2 regulations’, protect information generated at or received from federally assisted alcohol and drug abuse treatment programs.⁹³

All these laws, together, govern data privacy in one context: clinical health care.⁹⁴ AI/ML CDS tools, designed for that context, are subject to those laws. Data handlers operating AI systems elsewhere—for example, in research settings or in the broader surveillance society—are not bound by the same ethic of responsibility/fiduciary duties of health care providers. Outside clinical health care, control-over-information theory dominates although the ‘control’ it provides is often illusory.⁹⁵ The consents on which it relies are questionable: a recent survey found 97 per cent of Americans recall consenting to a company’s privacy policy but only 9 per cent reported reading the underlying privacy policies to which they agreed.⁹⁶ In 2017, one Wi-Fi company humorously proved ‘the lack of consumer awareness of what they are signing up to’ by inserting a clause in its privacy policy requiring 1000 hours of community service to clean toilets and relieve sewer blockages.⁹⁷ People consented.

Surveys show that people in the United States, Europe, and other technologically advanced societies yearn to control access to their data and believe consent and de-identification of data provide meaningful privacy protection.⁹⁸ Widely reported re-identification attacks over the past 15 years may have shaken their faith that de-identifying data protects privacy.⁹⁹ The response has been to double down on calls for consent: for example, by demanding consent for de-identified as well as identifiable data or passage of GDPR-themed legislation that many Americans hope might strengthen their consent rights. The effectiveness of consent as a privacy protection is rarely questioned despite being questionable.¹⁰⁰

Data handlers are major beneficiaries of control-over-information theory, which visits the burden of privacy protection not on the data handler but on autonomous individuals, who are expected to research privacy policies, shun risky data handlers, and protect their own privacy as best they can. Control-over-information theory is morally similar to telling the poor to pull themselves up by their bootstraps when, in reality, poverty (and privacy loss) may be systemic to modern information economies.¹⁰¹ Prof. Balkin’s first rule shifts the burden of data protection back to data handlers.

The White House Blueprint for an AI Bill of Rights, by dismissing context, overlooks the crucial role fiduciary duties and professional norms play in protecting privacy in the clinical care context. As a result, the Blueprint does not address the salient privacy challenge of AI-enabled health care. Introducing AI/ML CDS tools into the clinical treatment encounter exposes patient data to a new set of actors, such as software vendors and information service providers, who are not bound by the same fiduciary duties and confidentiality norms governing traditional health care providers. Federal action to strengthen patients’ consent rights would do little to protect patients’ privacy, without corresponding reforms to the framework of state laws and soft law norms that require parties who handle clinical data to be careful with it. Grand, sweeping statements about rights, such as those outlined in the Blueprint, belie the grinding work that lies ahead to craft tailored legal protections that can effectively protect rights in the various contexts where AI systems are going to be deployed. Oversimplifying a problem rarely solves it.

II.C. Rules at the Interface of Robots and Humans

Professors Balkin and Pasquale conceive ‘robots’ as including both mechanical robotic systems and stand-alone AI/ML software, and this article follows their convention.¹⁰² Robots without intelligence are morally equivalent to mechanical wind-up toys. The concern with robots is not their embodiment, but their ensoulment with something resembling (albeit faintly) human intelligence.¹⁰³ There are contexts where robotic embodiment might be legally significant—for example, if a robot has a fist that punches you in the face, as opposed to running an algorithm that merely advises a human intermediary to do so—but this is not such a context.

By design, AI/ML CDS tools offer recommendations to human health care professionals who can critique, reject, or follow the robot’s recommendations.¹⁰⁴ Software developers and vendors have little incentive to replace human professionals with robots, when keeping physicians in the loop provides well-insured decoys for tort lawyers to chase, possibly luring them off the scent of software vendors. Moreover, it lightens regulatory scrutiny at a time when, to date, only one fully autonomous AI/ML diagnostic tool has successfully emerged from the Food & Drug Administration’s (FDA’s) premarket review process.¹⁰⁵

This section judges rules for robots by their capacity to address three challenges a recent report by the U.S. Government Accountability Office and National Academies of Science, Engineering, and Medicine deems critical to the success of AI/ML CDS tools.¹⁰⁶ The challenges are: ‘protecting privacy’, ‘accessing high-quality data’ to train AI, and reducing ‘potential biases in data’ which can cause CDS tools to give unsafe recommendations for population subgroups underrepresented in the training data. To be fit for purpose in medical AI, any workable set of rules must, at a minimum, address those three challenges.

1. Managing externalities

Prof. Balkin’s second and third rules touch important concerns with AI in the modern surveillance society but seem less well tailored to AI-enabled health care. They address the potential for AI/ML tools to inflict negative externalities (harms) on people who are not involved in developing, using, or having their data processed by the software.¹⁰⁷ For example, a social media provider that manipulates its end users to swing a national election harms not only its users (who enjoy the benefits of the services provided) but everybody in the nation, including bystanders who gain no benefits and are only harmed.¹⁰⁸ Prof. Balkin’s second rule places users of AI/ML algorithms under ‘public duties’ to uninvolved parties ‘who are not clients, customers, and end-users’.¹⁰⁹ The third rule states, ‘The central public duty of algorithm users is to avoid externalizing the costs (harms) of their operations’.¹¹⁰

For AI/ML CDS tools, I argue, the central ethical concern is not externalities. Both the benefits and the harms are internalized, and in a way that could exacerbate health care inequities. The harms of AI/ML CDS software fall entirely on involved parties (‘clients, customers, and end-users’ of the software). Patients who are not treated using AI/ML CDS tools cannot be injured by their bad advice. Adverse impacts of operating CDS software flow to people connected, one way or another, to its operations. Those impacts include, for example, privacy risks for people whose data are used to train, validate, and operate the software; safety concerns for patients diagnosed and treated using CDS tools; and liability, reputational, and economic risks for healthcare systems and providers when CDS tools contribute to patient injuries. Uninvolved third parties will not experience these harms.

The benefits of AI/ML CDS tools are also internalized. The medical benefits flow to people included in the training data (either directly through inclusion of their own data, or virtually through inclusion of people ‘like’ themselves demographically, medically, socioeconomically, and in terms of race, gender, and environment).¹¹¹ These tools may not be fit for purpose in patients who were not well represented in the training data used when developing the tools.¹¹² Stated otherwise, CDS tools exhibit selection bias.¹¹³

Men of European ancestry were disproportionately included in the training data for many of today’s CDS tools, which tend to perform better for patients who are white males than for women and other racial and ethnic groups treated using the software.¹¹⁴ Common data processing practices, such as sorting data into a rigid male/female sex binary, can ‘erase’ patients whose sense of personal identity and gender differ from the sex assigned at birth.¹¹⁵ The resulting AI tools can overlook medically significant risks, such as an elevated risk of breast cancer in transgender males or of aortic aneurysm in transgender females.¹¹⁶ AI/ML tools developed in high-resource health systems serving privileged, well-insured patient populations can turn unreliable when rolled out to other health care facilities.¹¹⁷ Systems trained on data from individuals in high-income countries—that is, virtually all of today’s AI/ML CDS tools—may perform badly for users in low- and middle-income settings.¹¹⁸

Rules for managing externalities break down and are not the right rules for AI/ML CDS tools, where the devil lies in the internalities, not the externalities. Even so, Prof. Balkin’s rules offer a useful framing to guide policy. His crucial insight lies in setting distributional justice in the foreground of issues to be addressed by rules for robotic and AI systems. Justice was the neglected third Belmont principle in the ‘lopsided table’ of late 20th-century autonomy-centric bioethics.¹¹⁹ For AI/ML CDS tools, the right rules are those that mitigate selection bias, which is shaping up to be the central justice issue in AI-enabled health care.

Selection bias arises in health data sets because people willing to consent to secondary use of their health data differ medically from non-consenters—a fact that has been well documented for over 20 years.¹²⁰ Newer evidence suggests consenters also differ demographically, which suggests the medical errors would not be randomly distributed, but rather could fall in patterns resembling invidious health care discrimination.¹²¹ Post-Belmont bioethics, when it engaged with selection bias at all, tended to question whether it materially affects scientific results¹²² (or, when conceding it is material, discounted its ethical importance¹²³). These dismissive views no longer work in the age of AI-enabled clinical health care, where biased training data can exacerbate racial, ethnic, gender, and socioeconomic disparities in the quality of care.¹²⁴ If consent norms are feeding selection bias, that is a major ethical concern that can no longer be dismissed.

A complicating factor is that consent norms are just one of many contributors to a much larger problem with biased training data.¹²⁵ Much of the bias observed in today’s CDS tools reflects structural and systemic inequities in the broader economy and health care system.¹²⁶ People who are systematically denied access to health care do not generate clinical health data that can later be harnessed to train CDS tools. Such people are underrepresented in AI training data, not because of the consent norms, but because their data do not exist.

Other aspects selection bias, however, are influenced by the privacy norms a society adopts. For example, rules to anonymize data can strip away important details—such as residential zip codes—that might help detect non-inclusivity and bias.¹²⁷ Notice-and-consent norms can fuel future health inequity if, for example, people of color who have had bad prior experiences with biomedical research selectively decline to participate in AI training data.¹²⁸ Rule-dependent contributions to bias are a worthy focus for empirical analysis, critique, and reform.

Notice-and-consent privacy norms invite the same critique that has been leveled at legal liberalism more generally: they assume ‘all persons share certain “samenesses,” such as rationality and autonomy’ and they allegedly advance a ‘highly specific model of personhood that was constructed for a white male elite’.¹²⁹ When inclusion in training data requires consent, this can have unintended effects that oppress groups for whom the exercise of autonomy (eg, deciding whether share their data to train AI/ML CDS tools) carries group-specific risks (eg, deportation if the data later leak to immigration authorities, or prosecution for gynecological care after Dobbs v Jackson Women’s Health Organization).¹³⁰ The Belmont Report and the flowering of bioethical literature in its wake were largely the work of white scholars, and the field allegedly exudes white normativity. ¹³¹

AI/ML CDS tools challenge core assumptions of 20th century bioethics. For example, bioethics portrays people who contribute data to biomedical research as ‘altruistic’.¹³² Consenters who wrongly believe research will benefit their own health are said to have ‘therapeutic misconception’ and should receive clear notice, before consenting, that the benefits of research, if any, flow to others and not to data contributors.¹³³ AI/ML CDS tools upend this altruism narrative: the benefits of participating in training data are internalized, not externalized to others.¹³⁴

The altruism narrative, at heart, rests on an assumption of human sameness: contributing your data to research benefits others (and thus is ‘altruistic’) only if your data are representative of those other people you were hoping to help—that is, if humans share sameness. In clinical care contexts, people were never the same, as Sir William Osler remarked 130 years ago: ‘If it were not for the great variability among individuals, medicine might as well be a science and not an art’.¹³⁵ Modern studies of selection bias prove Osler was right.¹³⁶ In the mid-20th century, when bioethics was born, FDA’s clinical drug trials studied small groups of consenting volunteers, pretending they represented the entire human population and chalking it up to random ‘noise of human variability’ when FDA-approved products tested on largely male, largely white trial populations injured other patients.¹³⁷ The pretense is no longer defensible. The 21st-century health care system is serving an ever-more-diverse patient population and needs CDS tools trained on inclusive data representing all of us. Notice-and-consent privacy schemes, with their sameness presumptions, may be incompatible with that goal.

Prof. Balkin proposes an alternative: a duty-based privacy protection framework in which data handlers act as information fiduciaries. This aligns with the approach medical privacy law has embraced (even if the fiduciary duties it imposes may need strengthening for AI-enabled health care).¹³⁸ For AI software operating elsewhere in the information economy, perhaps control-over-information privacy theory is still appropriate; this article takes no position on that. However, if control-over-information theory truly is the ‘leading paradigm’ that guides the White House Blueprint for an AI Bill of Rights, the case for medical exceptionalism seems strong.¹³⁹

Privacy is only one of five values the Blueprint seeks to advance. The others are safety and effectiveness of AI systems (which requires broad access to high-quality data during system development, validation, and regulatory oversight of real-world system performance);¹⁴⁰ equity and non-discrimination (which requires inclusive data for ongoing monitoring of system performance and to afford remedies to persons injured by inequitable software);¹⁴¹ system transparency;¹⁴² and human intermediation and oversight.¹⁴³ In domains where selection bias is well-documented,¹⁴⁴ as in health care, the Blueprint’s notice-and-consent privacy scheme could undermine its other goals of consumer safety, justice, and equity.

The Blueprint resists balancing of public and private interests: system developers and operators should ‘seek your permission’ to use your data, and the social costs of restricting data access in this way are not part of its analysis.¹⁴⁵ The Blueprint acknowledges that certain contexts may require ‘exceptions’ to ‘balance competing public interests’ but focuses this remark on law enforcement, not health care.¹⁴⁶ Competing interests are not rare ‘exceptions’ but rather are the rule in any context where the use of personal data confers important benefits on society. Health care is one such context. The American Data Privacy and Protection Act that Congress has been developing recognizes this fact.¹⁴⁷ As currently drafted, it leaves in place the existing framework of state and federal medical privacy laws as well as various other sectoral privacy schemes addressing concerns in specific data-use contexts.¹⁴⁸ The Blueprint should embrace this same approach. Its notice-and-consent privacy scheme offers only weak privacy protection¹⁴⁹ while undermining other important social values that the Blueprint seeks to advance.

2. Etiquette in robot/human relations

Frank Pasquale’s rules address moral and legal concerns about the way robotic and AI/ML systems, especially the more-fully-autonomous ones, interface with human beings. He enunciates four laws of ‘complementarity, authenticity, cooperation, and attribution’ that encourage developers, operators, and users of AI/ML software to ‘develop policies that capitalize on human strengths…and bound the scope and intensity of conflict and regimentation in social life’.¹⁵⁰ The rules are: (i) ‘Robotic systems and AI should complement professionals, not replace them’;¹⁵¹ (ii) ‘Robotic systems and AI should not counterfeit humanity’;¹⁵² (iii) ‘Robotic systems and AI should not intensify zero-sum arms races’, whether in military, policing, credit-scoring, litigation, social control, or other contexts ‘where people compete for positional advantage’;¹⁵³ and (iv) ‘Robotic systems and AI must always indicate the identity of their creator(s), controller(s), and owner(s)’.¹⁵⁴

This article discusses AI/ML CDS tools which, by design, assist rather than replace healthcare professionals. This focus sidesteps many of Pasquale’s concerns, so this is not a good forum to do justice to his sparkling insights. Certainly, his concerns would be relevant if a CDS tool designed to complement healthcare professionals was applied in ways that counterfeit them, for example, by professionals using the tool off-label and passing its recommendations off as their own without informing the patient they are relying on it. State medical practice regulators, the FDA, healthcare accreditation bodies, and medical ethicists should develop policies to limit this and other potential misuses of AI/ML CDS tools.

Pasquale’s third law against using AI/ML to ‘intensify zero-sum arms races’ to enhance ‘positional advantage’ strikes closer to this article’s focus.¹⁵⁵ A simple Google search on the terms ‘zero-sum game’ and ‘health care’ yields many articles about zero-sum games between health insurers and hospitals, between hospitals seeking to cut costs and patients denied care, and between many other players in a complex, budget-constrained healthcare system. Prof. Cohen et al. cautioned, early on in 2014, that tools calibrated to optimize the health of a hospital’s entire patient population or to lower its costs might sacrifice the well-being of individual patients.¹⁵⁶ Transparency is crucial, so patients have notice whether AI/ML CDS tools are optimizing population health or their health.

A fairly diligent search found no sources exploring zero-sum games in the acquisition of data for AI/ML training datasets. This silence, I suggest, is because acquiring training data for AI/ML CDS software is a positive-sum game. The more people whose data are reflected in the training data, the better the AI performs for everyone (assuming, of course, equivalent data quality). Making training data more inclusive and representative of today’s diverse patient population does not take anything away from already-well-represented groups. Thus, adding gender-representative data to a male-dominated AI/ML CDS tool does not erode its performance for men; it just makes it perform better for all who are treated using the software. Making AI/ML training data more inclusive is a way AI/ML CDS software developers can honor Pasquale’s third law against zero-sum arms races. If that entails critiquing and modernizing popular notice-and-consent norms, Pasquale’s third law underscores the ethical duty to do so.

III. DIFFERENT RULES FOR SIMILAR INFERENCES

The notion that AI/ML medical software warrants special rules presumes medical AI is a clear category, distinct from other types of AI. The line between medical and non-medical software grows blurry in an age when AI tools processing non-medical data can draw inferences about people’s health. ¹⁵⁷ The Blueprint for an AI Bill of Rights treats ‘health’ data as an undifferentiated category, drawing no distinction between the clinical data licensed physicians develop while treating patients versus other health-inflected data trafficked in modern information economies.¹⁵⁸

This Part questions the Blueprint’s approach and distinguishes two categories of AI that process and produce health-related data. In this discussion, the phrases ‘clinical data’ and ‘clinical inferences’ denote health-related information generated and used to treat patients in traditional clinical care settings. The phrase ‘health inferences outside the clinical context’ (HIOCCs) refers to inferences about people’s health drawn by retailers, credit-scorers, employers, fitness trackers, at-home wellness monitoring devices, services, and other non-medical actors in a surveillance society. In terms of content, the information is at times quite similar.

The distinction between clinical health information and HIOCCs turns on intent. Although health-related, HIOCCs are not intended for use in diagnosing, treating, mitigating, or predicting disease or improving individual health. HIOCCs aid decision-making that affects people’s rights but usually not their health, at least not directly or intentionally. For example, HIOCCs guide decisions on loan eligibility, employment, advertising, and law enforcement, such as deciding to fine rather than imprison an elderly offender to avoid burdening prison health care with high-cost geriatric patients.¹⁵⁹ Such decisions can indirectly affect health—for example, if a home loan denial prevents someone from moving to a new neighborhood with better facilities for outdoor fitness activities—but the data are not intended to inform clinical health care but to assess credit risk.

Related questions of intent define FDA’s jurisdiction to regulate drugs and devices under the Food, Drug, and Cosmetic Act and limit the scope of federal laboratory oversight under the Clinical Laboratory Improvement Amendments of 1988 (CLIA).¹⁶⁰ Regulators and courts have successfully administered intent-based jurisdictional rules in those and other contexts, and FDA has even codified its approach for determining whether a product is intended for clinical use.¹⁶¹ This prior experience suggests that law is well able to wield the intent-based distinction proposed here.

III.A. All Data Are Health Data in an AI-Enabled World: The Case for Uniform, Content-Based Consent Norms

AI/ML tools at retail stores can infer—with considerable accuracy—whether a young woman is pregnant based on her pattern of purchasing ‘scent-free soap and extra-big bags of cotton balls, in addition to hand sanitizers and washcloths’.¹⁶² A 70-year-old woman or a male buying those same products would not trigger an inference of pregnancy, unless the software was trained to expect they might be expecting. In a surveillance society, all data are health data in the sense that they can be processed to draw health-related inferences, but the inferences are only as accurate as the training data that taught the algorithm what correlates with what.

The inference that Terry is pregnant seemingly deserves the same privacy protection, whether it came from a Target™ store’s AI algorithm, from an at-home pregnancy test, or from a clinical diagnostic test ordered by Terry’s health care provider.¹⁶³ Regardless of the source, the information content is similar and has the same sensitivity around disclosure and the same risks of stigmatization or discrimination, such as causing Terry not to get a job offer.

Almost 30 years ago, the Institute of Medicine (precursor of today’s National Academy of Medicine) called for uniform, content-based protections for health data.¹⁶⁴ Health data implicate historical and cultural taboos about the body and bodily functions.¹⁶⁵ They reveal behaviors (eg, substance abuse, sexual activity) and conditions (cancer, mental health, pregnancy) that people might prefer not to disclose.¹⁶⁶ These attributes fuel a popular expectation that health data should be subject to strong consent norms, wherever in the economy the data are stored.¹⁶⁷ The need for consent, by this view, reflects inherent data characteristics—for example, the type or substantive content of the data (eg, health-related data), the perceived sensitivity (eg, data on sexually transmitted diseases, as opposed to a patient’s blood type), or the data’s perceived re-identifiability (eg, genetic data).

The existing legal framework for health-related AI lacks uniform, content-based protections both for consumer safety or for privacy and other civil rights. The FDA and CLIA regulations, addressing data quality and consumer safety, only apply to AI intended for clinical use.¹⁶⁸ The HIPAA Privacy Rule and most state medical privacy laws take a sectoral approach, protecting health information generated or used in clinical health care settings (eg, hospitals, clinics, skilled nursing facilities, health insurers) but not in other contexts (including some contexts that seem health-care-related, such as data held by drug and device manufacturers).¹⁶⁹ When the HIPAA Privacy Rule does apply, it allows covered entities to share health data for a long list of enumerated purposes without individual consent.¹⁷⁰

The EU’s GDPR appears—at least to casual observers—to impose consistent consent norms on similar data, regardless of who controls the data.¹⁷¹ This perception arises because GDPR identifies ‘special categories’ of information that receive heightened protections.¹⁷² These categories include data that many people characterize as sensitive: data which, if disclosed, might subject a person to discrimination, stigmatization, embarrassment, stress, or other psycho-social harms. Data ‘concerning health’ are one of GDPR’s special categories.¹⁷³

Rather surprisingly, the HIPAA Privacy Rule is not particularly content-based. It protects individually identifiable ‘health information,’ but this is a legal term of art defined in the HIPAA statute, and it does not map neatly onto GDPR’s concept of data concerning health or onto people’s everyday conception of what is health-related.¹⁷⁴ HIPAA’s definition of health information is surprisingly content-neutral.¹⁷⁵ Data become health information, for purposes of HIPAA, if the data are ‘created or received by a health care provider…’ and broadly relate to ‘the past, present, or future physical or mental health or condition of an individual’ or to ‘the provision of health care’.¹⁷⁶ That ‘or’ means that information that is unrelated to ‘past, present, or future physical or mental health’ can still be HIPAA-protected health information simply by being communicated during a medical treatment encounter.

If Sally mentions to her doctor in the course of a clinical encounter that she read War and Peace last week, this fact technically becomes health information under HIPAA. Sometimes, that fact truly might be health-related (eg, marking Sally’s progress after recent eye surgery), but often it has no clinical significance. Still, if the information makes its way into Sally’s medical records, HIPAA treats it as health information based on the context: the information was received by Sally’s care provider and was related, even tangentially, to the provision of health care (eg, by enhancing physician/patient rapport and building trust). Context, more than content, is what causes facts to become health information under HIPAA and many other medical privacy laws.

With only one exception, the HIPAA Privacy Rule rejects the notion that some types of health information are sensitive and need heightened privacy protections. That exception is psychotherapy notes, which have special protections with strong consent norms, but this exception is narrow.¹⁷⁷ Otherwise, the U.S. Department of Health & Human Services’ (HHS) Office for Civil Rights (OCR), which administers HIPAA, has resisted pressure to grant special protections for data that many people ‘characterize as “sensitive,” including genomic, cancer, pregnancy, sexually transmitted disease, and mental health test results’.¹⁷⁸ HHS maintains that designating ‘“sensitive” and “non-sensitive” categories of health information would be a subjective endeavor and would not necessarily result in policies that are in the patient’s best interest’.¹⁷⁹

Some might disagree, but HHS’s view of data sensitivity has merit. Something as routine as a patient’s blood type might be far more sensitive than HIV or pregnancy status, if blood type reveals misattributed paternity (where the ostensible father is not really a person’s father, a revelation that can expose pediatric patients to a risk of nonsupport or domestic violence).¹⁸⁰ Sensitivity is subjective and highly context-dependent, making broad, content-based restrictions a dubious way to protect privacy. GDPR’s top-down sanctification of ‘special categories’ of data discounts the crucial roles subjectivity and context play in defining what is sensitive to each of us.

In the United States, HIOCCs and clinical inferences are subject to different consent norms. The same is true in many other parts of the world if one peruses the local medical privacy laws.¹⁸¹Apart from privacy, HIOCCs and clinical inferences are under different data-quality and consumer-safety regulations as well. There is a common-sense argument that uniform rules should apply to all AI systems that use or generate health-related data. The remaining sections offer a contrarian view, arguing against uniform rules.

III.B. The Weak Link between Consent and Personhood for Clinical Health Information

There are centuries-old norms about individual control over one’s own body. Unconsented touching of the human body has long been viewed as a battery (actionable in tort and under criminal law) as well as an offense to human dignity.¹⁸² This norm against unconsented touching of the body is not context-dependent: unconsented touching is a battery whether committed in a research setting, during clinical health care, in a retail store, or elsewhere.

This does not imply, however, that similar consent norms should apply to health data. Touching data about a person’s body is not the same thing as touching the person’s body. Bioethicists and medical privacy scholars have struggled (not always successfully) to theorize the individual’s right to exert strong control over health data describing oneself. The declared basis for this right is a jumble of assertions about respect for persons, autonomy, human dignity, and selfhood, but it is not quite clear how a consent-based privacy scheme advances those things.

The modern claim that controlling one’s health data is integral to selfhood calls to mind Charles Taylor’s lament about impoverished ontological accounts of the modern Self.¹⁸³ Modern culture, in his view, dethroned concepts of a higher or ‘good’ life, such as the Aristotelian notion that health and the human body are not ends in themselves but instruments for value-laden activities such as contemplation or action as a citizen.¹⁸⁴ Carl Schneider describes a ‘change in attitudes toward health—the elevation of health to a supreme value…taken as the primary moral value of our civilization’.¹⁸⁵ This view is epitomized by the fitness buff two researchers quoted as saying, ‘Health is everything for me, it is primordial, and I live for it’.¹⁸⁶ This account of selfhood portrays the human body and its condition as essential to selfhood. Even if you accept that portrayal—and has not the disability rights movement already debunked it?—this portrayal still does not explain how information about the body is essential to the Self.

Casting about for a theoretical framework, it has been tempting for bioethicists and medical privacy scholars to borrow the rich theorizations of why control over personal information matters in other contexts. A particularly enticing body of work explores ‘surveillance in post-industrial, digitally networked societies’ where personal information is routinely collected, stored, and made visible to others and then algorithmically transformed ‘in the active production of categories, narratives, and norms’ that can land us on no-fly lists, earn us a discount, tag us as risky or at-risk human beings, or cause a prospective employer to rule us out.¹⁸⁷ Some of these works try to explain why flows of personal information and a person’s ability to control them might influence individual self-development.¹⁸⁸

Often drawing on Michel Foucault’s work on prisons, studies of surveillance portray it as a technique of social discipline, in which access to our data is embedded in the very design of commercial and social institutions¹⁸⁹ and our data feed statistical analyses that norm us relative to one another as opposed universal norms.¹⁹⁰ Surveillance ‘operates on its subjects not only by the “normalized soul training” of Foucauldian theory, but also by seduction’ including ‘a cornucopia of benefits and pleasures, including price discounts, social status, and voyeuristic entertainment’ that induce people (willingly or unwittingly) to share data enabling the surveillance.¹⁹¹ These works state plausible mechanisms through which a lack of control over one’s data might really affect one’s Self.

Julie Cohen describes an additional strand of surveillance studies based on performance theory, which stresses that human identities are not fixed and invariant across all circumstances, but vary as people play different roles to different audiences.¹⁹² Stated otherwise, identity is a state rather than an enduring trait.¹⁹³ ‘The struggle for privacy is recast as the individual’s effort to assert multiplicity and resist “norming.”’¹⁹⁴ The Self to be protected here sounds of Walt Whitman’s internally contradictory, multitudinous Self: ‘Do I contradict myself?/Very well then I contradict myself./(I am large, I contain multitudes.)’.¹⁹⁵ Under this model of selfhood, it is plausible that the inability to control access to one’s data might damage the Self, by holding people to account for having performed a different Self 15 minutes ago.

Insights from these works enrich but also confuse discussions of medical privacy. Health care providers collect, store, and process personal data not as a means of social discipline or social control, but to try to cure our sick bodies to avert the untimely demise of our selfhood. Clinicians analyze our data and norm us relative to other patients, not as part of ‘normalized soul training’¹⁹⁶ but to assess, for example, whether we are different enough from other patients who had strokes while taking Rofecoxib (Vioxx™) that the drug might ease our pain without harming us.¹⁹⁷ Research that uses health data is defined, by federal regulations, as a systematic search for ‘generalizable knowledge’—a quest for universal norms of scientific truth rather than Foucauldian norming as a means of social control.¹⁹⁸

As for performance theory, multiplicity and duplicity are strongly discouraged as hindering the goals of clinical health care. If a mother with Münchausen Syndrome by Proxy wants to perform an identity in which her child has cancer when the child does not have cancer, we all want the pediatrician to detect and halt that performance of Mom’s selfhood, including by sharing information with other health care providers, with Children’s Protective Services, and with law enforcement.¹⁹⁹ The arguments that persuade us that consent is essential to selfhood in the broader surveillance society are not well matched to the goals of clinical health care.

III.C. Medical Privacy Law as a Contextual Privacy Scheme

The crux of a surveillance society, in the popular imagination, is that one’s data are being gathered, analyzed, scored, and potentially critiqued to one’s detriment almost everywhere. The problem is contextual but at a level of generality where the context of surveillance swallows almost every aspect of one’s life except when one stays home hiding under the bed unaccompanied by a geolocation-tracking smartphone.

Medical privacy laws, in contrast, are designed around one specific context, clinical health care.²⁰⁰ They implement Helen Nissenbaum’s concept of privacy as ‘contextual integrity.’²⁰¹ As she describes it, contextual integrity is not a definition of privacy or a theory of why privacy matters.²⁰² This sidesteps the need to theorize why consent to the release of a person’s health information is essential to selfhood. Every context has distinct ‘informational norms’, a set of expectations about what can appropriately be disclosed, used, or shared with others, and when, to whom, and on what terms.²⁰³ Contextual integrity—privacy—exists when actual information flows align with those norms.²⁰⁴

People and their data do not exist in ‘an undifferentiated social world, but as individuals in certain capacities (roles), in distinctive social contexts, such as health care, education, employment, the marketplace, and so on’.²⁰⁵ Information flows appropriate in one context might be inappropriate, and spark protests in the name of privacy, in others.²⁰⁶ It is inoffensive for a sales associate in the lingerie department to inquire about your bra size; the same question would be off-putting when applying for a home loan. A content-based conception of data privacy (for example, the idea that some data are inherently ‘sensitive,’ requiring uniform protection in all circumstances) fails, because the same piece of information might be permissible to share in one context but not in another.

This resonates with Prof. Balkin’s point about shifting the focus of privacy discourse ‘from the kind of information to the kinds of relationships that produce the information’ and constrain information flows within and beyond those relationships.²⁰⁷ Production of health data in a physician’s office takes place in a relationship of trust with (fairly) clear norms limiting what can be done with the data.²⁰⁸ In contrast, production of health data by a retailer processing customers’ purchase data takes place in a different relationship with fewer (if any) constraints on how the data might be used.

The social relationships that Prof. Balkin emphasizes are only one part of Nissenbaum’s contexts. A context also includes a set of ends and values that people serve by entering that context and forming those relationships.²⁰⁹ Clinical health care, for example, serves the ends of diagnosing and treating illnesses and protecting the public’s health—ends that most people consider important.²¹⁰ These ends and values (and not just the relationships people form to pursue them) affect privacy policy.²¹¹

By admitting contextual ends and values exist, Nissenbaum adds an element often missing in discussions of privacy policy. As Carter Snead notes, ‘American public bioethics is strongly anti-teleological. It does not recognize natural “ends” that guide understanding of the flourishing of the individual human’ and focuses instead on ‘the self-defining projects of the individual will’.²¹² Yet, in reality, contexts are teleological: they serve purposes.

The ends and values of a context influence its informational norms by informing the trade-offs that occasionally must be made, in any context, between the moral virtues of keeping data secret versus the moral virtues of data disclosure. Information disclosures that might be ethically justified to serve important ends and values in one context, such as clinical health care, might be inappropriate in other contexts where disclosures serve other and potentially less noble ends, such as serving unwanted advertisements to Internet users, denying someone a home loan, consigning people to no-fly lists, or targeting people for deportation. Each context has its own ends, and the ends justify—or fail to justify—a particular data disclosure.

This reasoning leads to a counterintuitive result: strong notice-and-consent privacy norms may be more justified in contexts that serve less important ends because, in such contexts, the moral value of unconsented disclosures is lower and less able to outweigh the individual’s interest in non-disclosure. By this view, the ethical justification for notice-and-consent privacy norms is inversely proportional to the importance of the ends and values a context serves. In clinical health care, unconsented access to my health data might be well justified, for example, if it helps doctors save another patient who is in immediate danger, or helps them track an epidemic that threatens public health. There is less (but still some) justification to use my health data without consent in biomedical research: research serves a valuable end—helping other patients—but does so with uncertainty and usually after years of delay. Continuing down the hierarchy of ends, unconsented data uses are least justified in the surveillance society writ large where others seek to use my health data for ends that are sometimes frivolous, unimportant, or downright harmful.

By this reckoning, notice-and-consent norms make more sense in the information society generally than in the special context of clinical health care. This utilitarian balancing goes against the weight of post-Belmont bioethical literature in which individual autonomy trumps competing moral principles.²¹³ Nissenbaum’s work is not a normative defense of balancing, but a pragmatic recognition that in every context, specific ends are being served and moral trade-offs do take place. Clinical health care is a context where, in practice, the moral value of data disclosures (for justice, beneficence, and non-maleficence) carries considerable weight.

III.D. The Role of Context in Establishing Rules for AI Tools

If medical and non-medical AI warrant different rules, this does not mean that HIOCCs deserve less protection than clinical inferences receive.²¹⁴ Three factors, discussed earlier, suggest that HIOCCs may deserve stronger consent requirements than law requires for clinical health information. Those factors were: (i) differences in the underlying social relationships and weaker fiduciary duties of data handlers outside the clinical context, (ii) differences in the values data disclosures serve, and (iii) concerns that consent norms can contribute to selection bias, and the gravity of those concerns in clinical contexts where biases can feed health care inequities.²¹⁵

A fourth factor is data quality. Medical privacy law embraces disclosure-friendly informational norms allowing unconsented disclosures of clinical information for a long list of purposes.²¹⁶ Similar informational norms might be highly inappropriate, allowing too much disclosure of health-related data, elsewhere in the surveillance society. Information generated in clinical care settings is subject to regulatory and professional standards aimed at ensuring data quality and accuracy. Clinical records sometimes do contain errors (even serious ones), but law creates strong incentives favoring accuracy. In contrast, HIOCCs derived from a person’s purchase data, Internet browsing history, or geolocation generally do not meet clinical standards of data quality. HIOCCs can be misleading and even (wildly) inaccurate. If stigmatization and discrimination based on people’s well-validated clinical health information is repugnant, then it is even more concerning to curtail people’s rights based on the dubious, low-quality surmises of poorly validated non-medical AI tools. Granting people a consent right alerts them that their HIOCCs are being circulated, possibly giving them a chance to dispute the information if it is inaccurate.

A fifth factor concerns ownership and control of inferences derived from health data. The HIPAA Privacy Rule and state medical privacy laws grant individuals a right of access not only to their raw personal data (eg, medical images and test results) but also medical opinions and inferences drawn from the data (eg, this person has cancer and is most likely to respond to Drug X).²¹⁷ HIPAA-covered entities must provide such access if patients request it.²¹⁸ Elsewhere, it can be hard for people to gain access to inferences about themselves, even when laws such as the EU’s GDPR provide access rights to data.²¹⁹ Under GDPR, people’s raw personal input data might be accessible, but the personal inferences often are not.²²⁰ Such policies recognize the legitimacy of competing claims to inferential knowledge. Inferences are not fully derivative of the input data on which they are based but also incorporate investments of effort, skill, and expertise. Under a Lockean ‘desert’ theory of ownership, people who draw inferences might have as legitimate a claim to an inference as does the person the inference describes.²²¹ Medical privacy law treats clinicians’ inferences as a sort of work-for-hire that the patient is entitled to know; this flows from healthcare providers’ fiduciary ‘duty of care’ to act in patients’ best interests, and from the fact that clinical inferences potentially affect patients’ health, which is the contextual value served by medical privacy laws.²²²

Similar handling of inferences would not necessarily be appropriate in general data processing contexts, where people’s raw personal data are processed to extract valuable insights and discoveries that might or might not have direct utility to the individual. As Lawrence Lessig points out, ‘No one spends money collecting these data to actually learn anything about you. They want to learn about people like you’.²²³ When that is true, data processors might have legitimate claims to their insights and discoveries, with individual having lesser interests in the inferences drawn from their data.

A final factor concerns concordance with traditional ways of making decisions about matters of public safety. Clinical health care obviously affects public safety, and the HIPAA Privacy Rule’s norms on unconsented data flows were designed to enable a set of ‘national priority’ activities that HHS determined were matters of public concern.²²⁴ The HIPAA Privacy Rule is a notice-and-consent privacy regulation only in a Blackstonian sense of consent: when a duly elected Congress empowers a federal regulator to decide which data uses are in the public’s interest, there is Blackstonian ‘consent of the people’—a consent produced by collective rather than individual decision-making.²²⁵

Collective consent is often used in deciding matters of public safety and concern. For example, modern societies routinely confide decisions about traffic safety to legislative decision-making, and once made, such decisions are binding on every member of the driving public with or without individual consent to observe the speed limit and obey traffic signals. Philosopher Charles Taylor acknowledges that one could argue that it violates individual autonomy to be forced to stop at traffic light, but ‘not in a serious political debate’.²²⁶ ‘[I]in such a case it is incorrect to speak of an infringement of freedom: the security and convenience of the walkers are in question, not freedom.’²²⁷ People perceive infringements on freedom ‘against a background of understanding that certain goals and activities are more significant than others’.²²⁸ This accords with Nissenbaum’s insight that contextual ends and values are crucial points to consider when setting privacy policy.²²⁹ In contrast, flows of HIOCCs in the broader surveillance society often are not matters of public safety or concern. While it goes against our intuition, this fact strengthens the case for individual control over HIOCCs.

IV. CONCLUSION

This article chose AI/ML CDS tools as a metaphor for many other AI tools that will raise context-dependent ethical and legal issues in the future. It identified contextual factors that justify exceptionalism in clinical data privacy. Despite continuing calls to embrace control-over-information theory, medical privacy law stubbornly clings to fiduciary duties of data handlers, rather than consent, as its main tool of privacy protection. While the overall approach seems sound, AI-enabled health care strains it by introducing new data handlers (such as commercial software developers) not bound by the framework of laws imposing fiduciary duties on traditional health care providers. Gap-filling reforms are needed to strengthen responsible data handling in AI systems intended for clinical use. ²³⁰

There is a strong case against extending medical privacy laws to non-medical AI systems that process or produce health-related data. Medical privacy laws allow and, in the case of individual access to data and inferences about oneself require, various disclosures. Those same norms might burden people’s privacy unjustifiably, or deny data processors the fruits of their own inferences, in other data processing contexts where the ends, values, data quality, and social relationships are different.

Beyond privacy, rules for robotic and AI systems must also promote justice, recognizing that the salient questions of justice are often context-dependent. The expanding role of AI in society requires us to right the lopsided bioethical table of the past 40 years that stressed autonomy to the near-exclusion of justice and equity. We are no longer in a strictly human-versus-human contest where the person exercising the most autonomy wins. A new contestant has entered the game—AI—and human-versus-AI conflicts will emerge. Humans must practice sticking together and working as a team in the waning hours before AI grows smarter than we are. Sticking together implies cultivating the neglected Belmont principles of beneficence and justice, while there is still time.