Legal infrastructure for transformative AI governance

Abstract

Most of our AI governance efforts focus on substance: What rules do we want in place? What limits or checks do we want to impose on AI development and deployment? But a key role for law is not only to establish substantive rules but also to establish legal and regulatory infrastructure to generate and implement rules. The transformative nature of AI calls especially for attention to building legal and regulatory frameworks. In this Perspective, I review three examples: the creation of registration regimes for frontier models; the creation of registration and identification regimes for autonomous agents; and the design of regulatory markets to facilitate a role for private companies to innovate and deliver AI regulatory services.

It was not long after the machine learning (ML) “Big Bang” of 2012, when University of Toronto researcher Geoff Hinton and his students demonstrated that a deep neural network could substantially outperform alternative approaches to image recognition (1), that calls for AI regulation began to emerge. AI researchers themselves were calling for study of the societal impacts of AI by 2015 (2) and a blockbuster investigative report by Pro Publica in 2016, claiming algorithmic bias in ML-based software used by US courts to evaluate the risk that a defendant seeking bail would reoffend (3), kicked off both regulatory and research agendas. When the European Union’s General Data Protection Regulation (GDPR) came into effect in May 2018, it contained perhaps the first regulation of AI: It required entities using automated decision-making systems (in, for example, credit or benefits determinations) to provide those affected with a right to obtain “meaningful information about the logic involved,” often referred to (not quite accurately) as a right to explanation (4). Principles and codes calling for responsible use of AI, focused primarily on fairness, transparency, privacy, and explanation, rapidly emerged from industry and civil society, with 84 statements issued globally by 2019 (5). The first major piece of AI regulation was introduced by the European Union in 2021. The next AI explosion, caused by the November 2022 release of the first generative AI made available to the general public (ChatGPT), triggered major last-minute revisions in the EU AI Act, global summits on AI safety, the establishment of AI safety institutes around the globe, and a sweeping Executive Order in the United States (later rescinded by a subsequent administration) laying out a framework for the use of generative AI in government and placing some requirements on producers of dual-use technologies. As of mid-2025, over 1,000 bills to regulate AI had been introduced at the state level in the United States. (6). Many of these continued to focus on the now decade-old concern about algorithmic bias, but the list of concerns had grown to include copyright, dangerous misuse, misinformation and deep fakes, manipulation, and loss of control. Major US lawsuits have also begun to address core features of generative AI, ranging from copyright claims challenging the legality of training large language models on copyrighted materials scraped from the internet to wrongful death suits laying responsibility for the suicides of young people at the feet of the companies building and deploying powerful chatbots.

Although the range of concerns has expanded significantly over the past few years to encompass not only individual harms like bias and loss of privacy but also systemic or existential risks such as a shift in the offense-defense balance in cybersecurity, threats to democracy and economic stability, concentration of economic and political power, increased capacity for bad actors to produce chemical or biological weapons, and even the displacement of human agency (7), debates about AI governance have yet to come to grips with the ways in which AI is likely to be truly transformative. Transformative AI is not, I believe, “normal technology” (8) that can and will be regulated effectively just through normal legal methods. If AI develops in the way those building it and funding it expect, it will disrupt how our legal systems, our methods of regulation, function (9). It will fundamentally shift how economic decisions are made, within firms and across the public and private sectors of society (10–12). And it is on the threshold of introducing entirely new autonomous agents into our economies and societies—agents that are not (yet) the subjects of legal systems (13, 14). Transformative AI demands more from law than the enactment of substantive rules about how AI functions. It demands innovation in legal infrastructure (15, 16), the rules, institutions, and processes by which those substantive rules are produced and implemented.

In this Perspective, I present a brief overview of three proposals for innovation in legal infrastructure that I have presented elsewhere and which can contribute to the development of effective AI governance: 1) registration regimes for advanced (frontier) AI models, 2) registration and identification regimes for newly emerging autonomous AI agents, and 3) regulatory markets that recruit greater private sector investment and innovation to the challenge of developing effective regulatory methods for AI. The laws that need to be enacted for these innovations for AI governance are constitutive and process-based rather than substantive. And they have been largely overlooked in AI governance debates.

Registration of Frontier Models: Legibility

Registration regimes are foundational pieces of legal infrastructure that enable courts and government regulators to function. By registration I mean a formal requirement that in order to participate legally in a jurisdiction, a person or entity, a piece of real or intellectual property, or a particular type of good must be known to a government agency of some kind. James Scott calls this the legibility on which the modern state is based (17). From the beginnings of the state, it was important for the development of taxation regimes for authorities to be able to produce official maps showing established plots of land and the people or entities responsible for paying taxes on that land and its output, replacing the highly local knowledge and overlapping uses of prestate societies. This in turn drove the need for a legally enshrined system for registering the identities of individuals and laws governing official names. The Qin dynasty in China imposed legal surnames on the population beginning in the 4th century B.C.E.; England and European jurisdictions followed a thousand years later; and colonial powers imposed surnames to rationalize colonial administration (17). Births, marriages, and deaths and transfers of rights in land needed to be officially recorded to track tax obligations to the state. Today, property registration plays a fundamental role in the financial arrangements that underpin market economies, allowing land and other real estate to serve as collateral for credit (18) and modern commercial law facilitates transactions secured against personal property such as machinery, inventory, or accounts receivable by creating public registries for such transactions (19).

Other registration regimes also emerged with the development of commercial trade. Early rulers and monarchs regulated trade by bestowing exclusive rights to engage in specific trades on corporate entities known as guilds, with the merchant guild system flourishing during the Commercial Revolution in medieval Europe (20). Registration of the corporate entity in a royal charter or decree underpinned these exclusive rights. The first companies were chartered by colonial powers to facilitate overseas trade and exploration (20). Guilds and companies, in turn, registered their members and owners, maintaining records of the individuals authorized to engage in a designated trade in a particular locality or to engage in stock transactions. This legal infrastructure implemented entry controls and a distribution of wealth both among guild or company members and with a ruler or government via taxes and other political benefits. Today registration requirements of authorized practitioners of a trade or profession abound, with registration required to practice law, medicine, securities brokerage, and more. Registration in these domains becomes a regulatory tool beyond entry control, enabling the loss of privileges to be used by the state as a penalty in the event a practitioner violates standards of practice.

As regulatory efforts expanded beyond the control of merchants and professionals, we see the emergence of registration regimes not just for individuals and entities but also for some products. Automobiles must be registered with a department of motor vehicles and issued a unique vehicle identification number (VIN). Establishments that manufacture, distribute, or import medical devices and the specific devices they market must be registered with the FDA (21). Pesticides must be registered with the EPA (22). Individual securities offerings must be registered with the SEC (23). In each case, registration is a critical step in establishing regulatory oversight.

Against this history and the essential role of registration to underwrite the effective functioning of regulatory regimes in the modern economy, it is striking that there are no registration regimes in place specifically with respect to AI in the United States.^* The only registration requirements that impact AI development are the ordinary business registration required of the entities that produce AI models and services and registration requirements that apply to products in which AI may be embedded such as medical devices.

Prior to the release of ChatGPT in November 2022, such an approach to registration may have been sufficient. AI regulation during this period followed the model of existing regulatory regimes. Regulation debates focused on specific applications of AI, such as autonomous vehicles or the use of AI in hiring or credit decisions. The targets of regulation were the developers and users (such as banks or employers) of AI systems. Most of these systems were built in a narrowly defined way for specific uses, largely based on supervised learning techniques in which an appropriate dataset is labeled to train a neural network to differentiate between, for example, a malignant and a benign tumor or a credit applicant likely to repay a loan and one likely to default. These uses of AI were highly legible to the state: They were embedded in products or processes by well-identified business or professional entities. These forms of AI did not significantly challenge the regulatory model. The original bill proposing the EU AI Act, introduced in 2021, modeled AI regulation on familiar methods of product safety regulation, and proposed a focus on use cases and risks (25).

The surprising advances associated with large language models post-2022, however, have significantly challenged the legibility of AI to the state and undermined the effort to simply extend existing product- and use-based regulatory approaches to AI. This challenge arises from a few central features.

First, these generative AI models^†, sometimes referred to as foundation models (26), are general-purpose technologies: The scope and scale of how they might be used and what might be built with and on top of them appears limitless. Yet the models are not simply science or math; they are themselves commercial artifacts being designed, sold, and delivered to users in a wide variety of ways. This means that there is potential need to regulate the base technology and not just the applications built with it. For comparison, we regulate another general-purpose technology—electricity—directly and not just the myriad products and uses that depend on electricity. Doing so ensures a stable and safe foundation on which those products and uses are built.

Second, we do not have scientific understanding of how or why these models work (27) or what capabilities may emerge as they grow in scale (28). Software built using ML is fundamentally different from conventional software. Conventional software is coded by a human programmer who writes down all the rules for how the software should process data. Even with highly complex software, such as the software that operates the autopilot on an airplane, we understand and can predict how it behaves; we can figure out what happened when it fails and trace that to code written by a human. But the software underlying AI models is written by a computer. The computer uses human-written learning algorithms to process training data and discover, through trial and error, the best mathematical operations to perform on the data to achieve a desired goal (such as predicting the next word in a sentence or attaching the correct label to an image). The software that results has hundreds of billions of steps in it and is not currently interpretable by humans. We do not know why, for example, if you fine-tune a publicly available model by showing it lots of examples of insecure computer code, it does not just get good at giving you insecure code when you give it a coding problem; it also advises you to handle your dissatisfaction with your spouse by hiring a hitman (29). The inscrutability of our AI models is then amplified by their general-purpose nature and ease of use: These features make it very difficult to predict what will happen when they are widely implemented in complex social and economic systems.

Third, the technologies of the most advanced models are still being built largely inside private companies, protected from public view by trade secret law, confidentiality agreements, and employee obligations to maintain company secrets. Even many “open” models—meaning models for which the weights or parameters of the neural network are shared publicly on the internet such as Meta’s Llama family of models—are built with methods and data sources that are hidden from the public. Moreover, even companies such as Mistral which release open-weight models continue to keep their most advanced models proprietary (https://mistral.ai/news/magistral). This may change over time, as increasingly powerful models like DeepSeek are truly open-sourced, but nonetheless our most advanced and widely used models are hard for those outside the frontier labs to analyze and predict. Academic and public sector researchers cannot reproduce the largest models and their training, fine-tuning, and inference pipelines in their labs and run experiments on them. This is in stark contrast with other complex technologies built inside private companies. Other technologies are generally embedded in products (such as automobiles or medical devices) that researchers and regulators can purchase, reverse engineer, and test, relying on public domain science (biology, chemistry, physics, engineering) to predict how the technology will perform and how its performance could be modified by various changes in the technology. Indeed, many of our most consequential technologies have historically been protected by patents, which require a public disclosure of how the technology works.

These features make it difficult for governments to figure out if, when, and how to regulate AI models themselves, separately from specific use cases such as medical devices or autonomous vehicles. For all the attention to governance of generative AI models post-ChatGPT—the EU AI Act, for example, was rapidly modified in the last stages to include requirements for generative AI, and several countries including the United Kingdom and the United States quickly created AI Safety Institutes to study generative AI risks—we still are significantly in the dark about the nature of real AI risks. Will generative AI models made available either with open weights or through APIs (like Gemini, Claude, and ChatGPT) fundamentally shift the offense-defense balance in cybersecurity, opening up new and unanticipated vulnerabilities? Can the models be used to significantly increase the risk that bad actors around the globe can produce chemical or biological weapons? Will AI companions engage in manipulation of children or adults, leading them to self-harm? Will models become uncontrollably autonomous and fundamentally misaligned with human interests? Will the delegation of financial transactions, through conventional banking or cryptocurrencies, destabilize financial markets? Will AI models responsible for pricing products and setting rents enable large-scale collusion or reduce the competitiveness of our markets? Will chatbot hallucinations and the capacity to generate deepfakes at scale disrupt social and political stability? These questions are the subject of intense debate, with leading scientists on either side. Yoshua Bengio and Geoff Hinton, awarded the 2018 Turing Prize for their foundational contributions to ML, are very sure that AI poses existential risks to humanity that governments should be acting immediately to address. Yann LeCun, who shared the 2018 prize, is very sure that these concerns are ridiculous and that regulation will unnecessarily throttle AI innovation (30). Whom are governments to believe?

We ideally want to ground defensible and effective policy on evidence (31). But the depth and pace of innovation is unlike anything we have dealt with historically. We simply do not know and cannot predict how these models will be used and how they will perform at scale and when integrated in almost limitless ways into our economic, social, and political systems. To a large extent, the only people who might know the answers to some of these questions are those bound to secrecy within AI technology companies. What governments and the public, and even most academic scientists, currently know about how these phenomenal new technologies are built and how they behave is largely limited to what private technology companies have chosen voluntarily to disclose.

This legibility problem is why, in July 2023, Tino Cuéllar, Tim O’Reilly, and I proposed the creation of a registration regime for large AI models (32, 33). On the theory that you cannot regulate what you cannot see, we proposed that developers seeking to deploy a model in the country be required to register their most advanced models with a national registry. We suggested an initial threshold just beyond the then-largest model available (OpenAI’s GPT4). The essential point is not size of the model per se but rather the notion of the “frontier” model, literally the one that resides in the unexplored territory just beyond the boundary of the legible state. We fully anticipated that this threshold would evolve but it was proposed based not on an assessment of where risks from models begin but rather where the legibility problem for the state begins.

Registration under our proposal would not require meeting any required standards or conducting any particular tests, just disclosure: the size of the model as measured in terms of the compute and quantity of data used to train it and the number of its parameters (weights), the nature and sources of the data used for training, what tests the developer has conducted to evaluate the behavior and safety of the model, and what the developer knows about the model’s capabilities and risks. Disclosure should be reasonably comprehensive and subject to penalties for lack of completeness or fraud, but ideally this should not be a major burden on developers, large or small. Moreover, it implements good organizational practices within AI companies, requiring them to maintain good internal knowledge of what they are building and what they know about it. This is essential for such novel, opaque, and fast-moving technology. Of course, the burden of these disclosures to the government needs to be weighed against the benefits (34); the goal here is to establish a minimal set of disclosure requirements to build effective future regulation.

Disclosure under this approach would be confidential to the government only. Although there are some features of models that we may well want shared publicly, it is important to recognize that the goal of the registration regime is not public transparency (which is the subject of other regulatory proposals (34) which can be adopted in parallel) but rather legibility to the state. As with other regulatory contexts (such as tax) government can be afforded access to commercial details to perform its regulatory function without a determination that all information should be in the public domain. The registry should be staffed in such a way that disclosure serves the purpose of building government capacity to understand where the technology is and to begin to gain the technical expertise and evidence base needed. Ideally the registry serves as the seed crystal for any regulatory agencies that evidence suggests should be created. With disclosures from all developers seeking to deploy in the national market, government experts would have a landscape view of technological development and can begin to evaluate what the systemic effects of this transformational technology might be.

Importantly, we propose that this registration scheme operates like the registration required of workers and business corporations: Registration should be a legal requirement not only for sellers of AI but also for buyers. Just as employers and banks are subject to penalties if they fail to check that a prospective employee is authorized to work in the jurisdiction or that a business seeking to open a bank account or take out a loan is properly registered with the state, so too would the (commercial) purchasers of models and their services be subject to penalties if they failed to verify valid model registration. This approach to enforcement creates a market incentive for model providers to comply and embeds compliance monitoring in a broad decentralized network, reducing the burden on the state. It also facilitates regulation of foreign developers: Registration becomes a requirement of market access, as a practical matter. Moreover, because buyers of models and model services would be required to verify registration, registration can operate as an “off-switch” (35): In the event significant enough risks are detected (for example, if one developer discloses dangerous capabilities other developers have not yet evaluated), suspension of registration can rapidly halt deployment of a model. There are technical design questions to explore here. A registration regime likely requires creating the technical capacity for a buyer/user to verify the registration status of a model on an ongoing basis—potentially at each call of a model on an API for example. There are also legal design questions: Should registration be checked even with purely internal deployment within a business, as when a business downloads an open-weights model and uses it for its own business processes? This depends on the nature of the risks associated with such models. Currently, open weights models do not meet the definition of frontier model as suggested in (26) as they are considerably smaller than GPT4.

As AI technology evolves, we may discover that increasingly capable smaller models require regulation, but that is the point of registration: It is basic legal infrastructure on which any further regulation of models can be built as needed. We do not need to resolve the hotly debated question of the nature of the risks associated with our most powerful models to recognize the wisdom of increasing the legibility to government of what is happening in frontier AI development. Indeed, model registration is what we need to thoughtfully resolve those debates. Nor would or should all regulation occur through the registry; recognition of risks within an expert registry can prompt lawmaking in other appropriate agencies and government departments. Registry expertise can also build out to foster greater expertise in those other agencies and departments. Registration is an important legal regime that allows us to build appropriate and effective substantive AI regulation.

Registration of AI Agents: Accountability

Registration regimes not only underpin the functioning of the regulatory state by making entities legible to governments, they also form the base layer of the legal system, making entities legible to the legal system and thereby securing our accountability regimes. People and organizations file lawsuits to hold those they interact with accountable for violations of common law (property, contract, tort) or statutory rights. Individuals must ordinarily file lawsuits in their real (registered) names. A lawsuit can only be started against a defendant in a court over whom the court has personal jurisdiction. For an individual this is often determined by their legal (registered) home address and service of process to that address generally suffices to initiate a lawsuit. A business entity’s capacity to sue and be sued in court rests on its registration of the business in the state, providing the name of an agent and an address at which legal documents may be served.

The functioning of courts, in turn, is essential to the functioning of markets. At its most basic level, market exchange rests on the enforceability of property rights and contracts. If generative AI was the important shift in the trajectory of AI development that challenged the legibility of AI to the state, the emergence of AI agents is the shift that challenges the legibility of AI to the courts. AI developers began announcing in early 2025 that this would be the “year of AI agents,” the beginning of the “agentic era” in AI (36, 37). Whether and in what form the technology of AI agents will become feasible is still a question mark. But what is clear is that if we do see these predictions come to pass, we will face a host of novel questions about the accountability for the actions taken by AI agents, often far removed from the instructions and oversight of a human. Questions of accountability for personal injury (e.g. torts) caused by AI agents have been with us for several years, notably with respect to autonomous vehicles (38) and medical applications (39, 40). In this section, I focus on accountability in economic transactions and relationships and argue that we lack the basic registration infrastructure that agents will require to function effectively in, and avoid major disruption of, our markets (41).

An AI agent is an AI system that is capable of taking in fairly general instructions, coming up with a plan of action, and then acting autonomously on that plan using tools that impact the environment such as by sending an email, executing a contract, standing up a website, or running computer code to operate a system or machine. Mustafa Suleyman, a cofounder of DeepMind and subsequently CEO of Microsoft AI, has given a graphic description of what an advanced AI agent could do, namely pass what he called the “Modern Turing Test:^‡”

• Successfully act on this instruction: “Go make $1 million on a retail web platform in a few months with just a $100,000 investment (42).”

It is important to pause on what an AI system capable of passing the test could be doing in the world (43): researching consumer product markets, perhaps by commissioning a market survey firm or launching an online survey; researching any consumer product regulation or guidance that might apply, adjusting design or marketing choices to comply (or not) with requirements; entering into supply contracts; contracting for logistics and warehousing, including terms governing risk allocation and insurance; hiring any humans that might be needed to accomplish particular tasks; setting prices; arranging for orders to be received and fulfilled. Other visions of AI agents are similarly or even more comprehensive: OpenAI defines the “agents” stage of AI development as consisting of “AI systems that can spend several days taking actions on a user’s behalf” and contemplates the capacity for AI systems to eventually run entire organizations (44).

This is a vision of artificial agents deeply engaged with the economic system and, consequently, the legal system. But who exactly is it that is engaged? Who is the party to the contract between the business and a supplier or customer? Who is liable for regulations violated, or intellectual property rights infringed? Who is responsible for employee relations? Who is entitled to sue for fraudulent representations or anticompetitive behavior that harm the business the agent is engaged in?

The glib answer is the user that instructed the agent and sent it off to do things in the world. But how will the individuals, businesses, and government regulators who interact with the agent identify the user behind the agent? The agents that were available as of mid-2025, such as OpenAI’s Operator, required constant babysitting by the user: No transactions, such as a purchase of groceries or an airline ticket, could happen without the user entering their own name and credit card details. But the vision of the AI companies, and the end to which they are directing billions of dollars in investment, is precisely to take the user out of the loop in an extensive way, so that agents can spend “several days taking actions on a user’s behalf.”

The missing ingredient for all this to work and make sense is registration (45). In fact, we already have a well-developed registration regime to handle some artificial actors, namely corporations. The registration we require of entities doing business in a state is precisely addressed to the question of how to effectively initiate a lawsuit to enforce a claim against the entity—where to serve the legal papers and who is designated as the official “agent” of the corporation in the state. This registration requirement both provides a public record of who is legally responsible for responding to filed complaints and obviates the need to track down the owners or shareholders behind an organization. We do not need to identify the “user” who put the business entity in motion; we just need to consult the formal records to activate the legal accountability process.

If AI systems are going to operate with high degrees of autonomy in our markets, they too will need to have public, durable, and traceable identification, just as other market participants do. This is accomplished by law creating a public regime of identification and legal requirements for agents to be registered in these regimes.^§ If a government elects a liability regime governing agents (a question of substantive law) in which the user instructing and deploying an agent or the developer serving the foundation model on which the agent is built is strictly liable for any actions taken by that agent (contracts signed, regulatory documents submitted, design and pricing choices made, etc.) then legislatures will have to pass laws requiring any agent to be publicly associated with an accountable human or business entity. Those interacting with the agent seeking to enforce a contract or sue for damages will have to be able to reliably determine on whom they need to serve legal papers, just as businesses must file registration documents with the state showing a legal address and registered agent authorized to receive service of process.

There is a reasonable chance, however, that if the vision for AI agents comes to pass, then for our markets and legal accountability regimes to function effectively, agents will have to have the capacity to sue and be sued directly. This is the definition of legal personhood, which dates back as far as the Roman Republic, with municipalities and voluntary organizations such as the Catholic Church recognized as entities capable of owning property and entering into contracts and thus of suing and being sued independent of the individuals who created and controlled them (46). And as we have already seen, the idea was revived in medieval Europe with the chartering of guilds and regulated companies. The reasons that substantive law for AI agents may well evolve in this direction are comparable to the reasons the law evolved in this way for corporations: Transactions that must be constantly traced back to the assets and accountability of individual participants in a commercial association incur high transaction costs. Information about the relationship between actors is hard to come by. There are agency problems between the individual participants, putting stress on the reasonableness of holding one responsible for the unforeseeable actions of another (47, 48). If AI agency does evolve in this direction, with a deliberate substantive choice to establish AI agents as legal persons capable of owning property and entering into contracts in their own “name,” then clearly this will require a registration scheme that creates a public and formal record of the identity of individual agents and associates their assets and legal rights and obligations with them. This will be necessary for them to sue and be sued, the access to courts and the legal system needed to ground their economic activity.^¶

Finally, a registration regime for AI agents is essential infrastructure for the very likely case in which we develop substantive rules about how AI agents are authorized to act and what characteristics they must have (how they must have been trained) to be safe and efficient participants in our markets and societies. As noted above, if a registration regime is enforced by creating a legal obligation for counterparties—those transacting with an agent—to verify the validity of the agent’s registration, then registration can operate as a form of “off-switch.” AI agents that do not meet substantive requirements for how such agents are trained, or which have violated legal rules governing their conduct, can have their registration revoked. That can quickly shut off the economic value of an agent, creating an incentive both for appropriate training and for lawful conduct.

Regulatory Markets: Regulatory Innovation

A major challenge faced of appropriate governance for AI is the growing chasm between industry and government in terms of technical knowledge and the capacity for innovation and adaptation. The “pacing” problem facing governments, which move slowly relatively to industry in an era of rapid technological change, has been understood for a long time (49). But the knowledge problem with respect to AI is novel. It is not (just) the challenge of hiring sufficiently expert technical staff in the civil service. More fundamentally, it is a function of my earlier observation: Most of the knowledge about what AI models are being built, how they function, and what their capabilities and risks are, is held inside private technology companies. Moreover, there is no science to guide us in predicting how today’s massive and opaque AI models will behave. Although some evaluations of the behavior of AI models are possible with only black-box access, in many cases evaluation requires access to the models and their production processes and the capacity to experiment with them.^# This is a relatively new problem for government regulation: Earlier generations of technology have been built on a lot of public science (chemistry, biology, engineering) and patented methods that are published. In some cases (such as military technology) government has funded and participated in the development of the technology. Where methods have remained protected by trade secret, they have been embedded in objects (goods, machines) that could be tested or reverse engineered by government and academic researchers: Automobiles and medical devices can be purchased and put through crash tests or clinical trials; academic researchers can produce and test alternative automobile and device designs. But the sheer cost to produce frontier models—today’s models cost tens of millions of dollars to train and more to finetune and serve—currently puts them out of reach to academic researchers. And scale matters: We do not know why, but the same training methods produce unexpected capabilities simply with increased scale (28). Moreover, understanding how these models behave when they are run in response to real human queries (a process called inference) requires access to the vast quantity of data collected from operating these models with millions of users. The cost of research independent of industry is a major hurdle for public oversight.

Because of these challenges, as I have advocated elsewhere (16, 51), another key infrastructural innovation we need for AI governance is the creation of new regimes for producing regulation.

Conventional regulation is structured as command-and-control regulation. Government enacts laws that specify what industry is required to do to avoid administrative penalties or civil remedies. In the context of AI governance, command-and-control regulation takes the form of, for example, requiring an AI developer to implement a risk-management system (a key feature of the EU AI Act) or requiring that a generative AI model answer an established percentage of questions in a standard question bank “correctly” (a component of China’s 2024 draft security standard governing the licensing of AI models) (52).

Much of the AI governance effort around the world, however, has struggled to develop conventional regulation (34). The EU AI Act relies heavily on private industry standard-setting bodies to supply much of the detail of what is required of, for example, a risk-management system. One of the reasons there has been little regulation enacted, in fact, is that the complexity and opacity of AI technologies and the speed with which they are advancing makes it almost impossible to specify in any detail what developers should be required to do. The Biden Executive Order on AI, issued in October of 2023 (and repealed by the Trump administration in January 2025), contained minimal directives to industry, largely confined to informing the government if the developer was training a model larger than a specified threshold and disclosing, for example, the results of red-teaming tests that evaluate the potential of a model to lower the barrier to entry for the development of biological weapons or engage in self-replication or propagation.^|| The E.O. did not specify what red-teaming tests to conduct.

By the turn of the 21st century, the obstacles to command-and-control regulation for fast-paced complex technologies had generated “new governance” techniques to supplement conventional approaches in many domains. These techniques include voluntary self-regulation, perhaps with the involvement of government in generating codes of practice; self-auditing; management-based approaches, whereby companies develop risk-management plans and are required to demonstrate compliance with their own plan; and performance-based approaches that set outcomes that companies are required to meet using whatever techniques they deem appropriate (53).

All these methods are evident to some extent in current approaches to AI governance. But also evident are two critical deficits in these methods (54). One is a democratic deficit: Private actors (AI technology companies themselves and industry-led standard-setting bodies) have been left to decide the level of acceptable risk, with little input or oversight from governments. This is not a challenge in domains where risks are well understood—such as food safety (55)—but it is a massive challenge with respect to a highly novel, general-purpose technology that can impact almost every economic activity. The second is a technical deficit: Unlike conventional regulatory settings, we lack established science for both robust evaluation of the risks posed by a particular AI system and for the methods to reliably mitigate or eliminate risks (56). This means that existing approaches to AI governance have yet to produce clear technical guidance for what they must do to comply with legal requirements. Compare this to, say, pharmaceutical regulation, where drug developers have science to draw on to know 1) what risks to evaluate for and 2) how to conduct and do statistics based on clinical trials. Current red-teaming methods, in contrast, are largely ad hoc and constantly shifting in response to the evolution of models, training methods, and capabilities. This is why governments have been unable to specify technical details about how red-teaming tests are to be conducted and what is an acceptable result. Indeed, even in the far more familiar domain of algorithmic bias, with a long history of established measurement methods and legal standards for what constitutes unlawful discrimination against protected groups, governments have not specified the technical details of what model developers and users are required to do to avoid liability (57).

Democratic and technical deficits are related. Governments recognize that they lack the capacity for supplying technical requirements, especially at a rate that adapts effectively to a rapidly evolving technology and across the vast landscape of domains in which the technology is used. As a result, they leave much of the job of establishing technical requirements to industry and industry-led standard-setting bodies.

Together with Jack Clark, cofounder of Anthropic, I have proposed a different resolution to the technical and democratic challenges that loom so large for AI governance: regulatory markets. This is a novel regulatory approach that builds on new governance techniques but goes beyond models we currently see employed by governments. Building regulatory markets requires the enactment of the legal provisions to create and oversee these markets.

A regulatory market consists, schematically, of three types of actors: governments, target companies (such as AI developers), and licensed private sector entities, which we call regulatory service providers (RSPs) (58). The model builds on performance-based regulation: Instead of specifying the technical requirements of compliance, governments specify metrics and principles representing outcomes in a given domain. In the biorisk domain, for example, the outcome might be that an AI model does not increase the risk that a nonstate actor can develop a bioweapon above the risk generated from existing non-AI methods and tools. In the domain of algorithmic bias in hiring an outcome might be that an AI hiring tool does not generate disparate impact that would violate Title VII if litigated fully and fairly. Where possible, metrics might be developed: A government could select one of the (many and mutually incompatible) statistical measures of algorithmic bias, for example, and require that measured bias fall below a designated threshold; accident rates for autonomous vehicles could be required to fall below current rates observed with human drivers; the probability of chatbot hallucinations could be required to fall below a designated level.

The novel piece of the model is that the outcomes the government sets are not established directly as compliance standards for target companies. Rather they are used as the criteria for licensing and overseeing multiple, competitive, third-party private sector regulatory providers. These RSPs assume the role of translating outcomes into technical and organizational requirements that their customers—the target companies that elect a given RSP as their service provider—must meet. This approach is a refinement of the original performance-based regulatory model (59), which leaves the task of translating government-set outcomes into technical and organizational requirements to target companies themselves. In the regulatory markets model, this task is moved into an independent and regulated third-party sector. In the biorisk domain, for example, independent RSPs could develop protocols for red-teaming to assess whether a model raises the risk of bioweapons development above the government-set risk level and then conduct evaluations of target companies’ models and systems using those protocols. This “contracting out” of the task to independent actors is already underway: Although AI developers are engaging in their own red-teaming tests, they are also contracting with independent third parties to do red-teaming (60, 61). The innovation in this model is that these third parties would be regulated by the government; they would have to continue to demonstrate that their regulatory methods meet the outcome criteria set by government. Failure to do so would put them at risk of license suspension or loss. The fact that there are multiple licensed RSPs means that license suspension or loss can be a credible threat, underwriting the RSP’s incentive to ensure that its regulatory methods are a faithful implementation of government-set outcomes. This government oversight addresses the democratic deficit of the extensive delegation to private actors (including AI developers themselves) evident in current AI governance efforts.

Provided a target company is in good standing with a licensed RSP, it is in good standing with the government. Failure to engage a licensed RSP and maintain good standing would result in penalties. The efficacy of this approach then turns on the fidelity of RSPs and the efficacy of the regulation of those regulators: If the RSPs comply with government-set outcomes, their customers are achieving government-set outcomes. Similar schemes are familiar from the organization of various professions. The Financial Industry Regulatory Authority (FINRA), for example, is a private sector membership organization composed of securities broker-dealers that is authorized to regulate its members. It creates rules and procedures that broker-dealer firms must follow and which it can enforce with fines, suspension, and expulsion from the organization and hence the market. Unlike other professional bodies (such as those regulating lawyers and doctors) FINRA is regulated by a government agency, the Securities and Exchange Commission (SEC). Any rules FINRA adopts must be approved by the SEC and the SEC has general oversight authority of how FINRA is governed and how it conducts its rulemaking and enforcement activities (62). While the SEC continues to enact and enforce its own rules governing broker-dealers, it does not enforce FINRA rules. For example, FINRA Rule 3110 requires members to establish supervisory systems to ensure compliance with securities laws and regulations. If a broker-dealer fails to implement a supervisory system consistent with FINRA requirements, FINRA, not the SEC, takes enforcement action. If a broker-dealer violates securities laws, the SEC enforces.

A key difference between the regulatory markets model and existing regimes relying on regulated private sector regulators like FINRA is the “markets” component: Government licenses multiple RSPs that compete to sign up target companies.^** This is an essential component to address the technical deficit that stymies AI governance. The insight is this: determining the technical and organizational requirements necessary to achieve government-set outcomes in the development and deployment of complex and fast-moving AI technology is itself a massive scientific, organizational, and technology challenge. It will require significant investment in R&D directed to this challenge. It is not just a matter of drafting the right legislative language. It is a matter of inventing new technologies to evaluate, monitor, and intervene on AI technologies, including AI to regulate AI. The “science of AI evaluation” needs to be built (56), as does the art and science of implementing organizational and technical controls that shape AI development and deployment. These are new technologies (using that term to mean both things like software and things like organizational processes) that require financial and human capital directed to this type of innovation (16). The virtue of a market for RSPs is the creation of an industry dedicated to the development of this regulatory technology.

Robust development of regulatory technology for AI requires the harnessing of the private sector to attract sufficient financial and human capital. RSPs could be nonprofit entities that fund their investments and operations with a combination of fee-for-service and philanthropy—much as nonprofit hospitals and universities do. But they could also be for-profit entities, attracting venture capital in their start-up phases and accessing public and private capital markets as they scale. The key is that the system does not ask the public to rely on the motives of the owners and managers of these entities; these are regulated entities that lose their capacity to compete if they fail to deliver on their obligation to meet government-set outcomes. The integrity of the model rests entirely on the capacity of government to regulate RSPs effectively.

Effective regulation of RSPs would not be an easy task; but it is one that governments should be more capable of accomplishing than they are of directly regulating AI developers and users. This is the basic stance of the performance-based approach to regulation: Governments can evaluate performance (outcomes) even if they do not know what the right technology or process is for achieving that performance. Undoubtedly government will need technical capacity to carry out oversight of RSPs but a different kind or perhaps amount of technical capacity than if they are regulating directly. RSPs are also likely to be more legible to governments: They operate under license and governments can demand whatever data-sharing and reporting they feel is needed for oversight. There are also likely to be fewer RSPs than there are target companies, and these RSPs have an incentive to ensure that competitor RSPs are not misrepresenting capabilities or risks to the government.

Of course, any system of regulation is at risk of regulatory capture and fraud. The regulatory markets model reshapes the incentive and information structure of the capture problem. Politicians and public regulators have well-documented incentives and operate under informational burdens that tempt them to put corporate interest ahead of public interest (63, 64): campaign finance, the revolving door between government and industry, incomplete information, informational overloads, and insufficient expertise, not to mention straight-up corruption and bribery. Private RSPs indeed face a corporate (profit or nonprofit) incentive to make their services attractive to a larger set of consumers, driving up their market share; in fact, that is the source of an essential benefit of turning to private regulators, that they have an incentive to produce cost-effective and less burdensome regulation. But there is a limit to this incentive: Their market share drops to zero if government determines that they have failed to achieve the performance goals set for them. Or if they have engaged in fraud or accepted bribes. Moreover, the capacity for expertise, funded by market revenues and investment, puts private regulators less at risk of being misled or overwhelmed than public regulators whose budgets are controlled by politics rather than markets. The claim is not that private regulators will not be subject to regulatory capture, but rather that the likelihood of capture is probably not higher and may well be lower than with public regulators.

Implementing a robust regulatory market for AI requires careful legal design: To establish the licensing regime governing RSPs, requirements necessary to ensure that RSPs are competitive (such as data portability that enable target companies to switch RSPs relatively smoothly) and that there is sufficient scale in a given domain for competition, conflict of interest rules to ensure RSPs are independent of the entities they regulate, mandates for AI target companies to engage an RSP, etc. As a matter of AI governance, the shift is from a focus on what substantive requirements to place on AI developers and users to a focus on the creation of the legal infrastructure that can best foster the development of effective regulation.

Conclusion

In earlier work about the challenges of building regulation to meet the needs of a technologically advanced and global market economy (15), I adopted the term “legal infrastructure” to describe the constellation of actors, markets, norms, and institutions that produce legal outcomes. “Infrastructure” means the structure that lies beneath other structures. It is often hidden and taken-for-granted—until it stops working well: The bridge is out; the internet is down. Today’s debates about AI governance, focused as they are on the substantive question of whether and how AI development should be regulated, take for granted the hidden legal infrastructure that produces and implements legal and regulatory regimes in the first place. It takes for granted the laws we need to enact to build the, very different, legal and regulatory regimes we will need for transformative AI. In this essay, I have laid out three examples of key innovations we need in legal infrastructure to build effective AI governance: registration regimes for frontier models, registration regimes for AI agents, and regulatory markets to resolve both the technical and democratic deficits that currently bedevil AI governance efforts. There are other important innovations to explore (34). For legal scholars and regulators interested in AI, there are important contributions to be made in focusing attention on how to build the legal infrastructure that will more robustly put us on the path to ensuring that the AI transformation underway promotes human welfare.

Data, Materials, and Software Availability

There are no data underlying this work.