Weak artificial intelligence safety regulation can backfire and produce more dangerous AI systems than no regulation at all, a new study argues. The peer-reviewed paper, published in the Proceedings of the National Academy of Sciences, uses theoretical economics and game theory to understand how different regulatory approaches affect safety investments across the AI supply chain. The authors, from Cornell University and Carnegie Mellon University, conclude that strict standards aimed at every layer of the AI industry — not just the applications that reach consumers — are the most reliable way to ensure responsible development.

Key facts from the study

• The study applies game theory to AI regulation, specifically the classic prisoner’s dilemma.
• Researchers modeled the relationship between general-purpose AI providers and downstream specialists.
• They found that weak regulation focused on downstream firms encourages free-riding by model developers.
• The paper argues that safety and commercial utility can both improve under strict, well-targeted regulation.
• Principal author Benjamin Laufer emphasizes the need to regulate the whole AI supply chain, not a single provider.

Why regulating only downstream companies can fail

At first glance, regulating the obvious user-facing applications seems logical. A company using an AI model to make medical diagnoses, approve loans, or power customer service chatbots is directly accountable for the harm that the technology might cause. But the researchers say this approach misses an upstream dynamic. When governments impose safety obligations on downstream companies while leaving general-purpose AI developers untouched, those developers have little financial incentive to invest in safety. They can instead assume that the downstream specialist will be responsible for testing, monitoring, and fixing any problems before the product reaches users.

That creates what Laufer calls a “free-riding behavior.” The general-purpose AI provider offloads the safety burden onto the downstream company. For example, a large AI lab that develops a foundation model can decide to skip third-party audits, red-teaming, or robust documentation, because the regulation does not require the lab to do them. The lab knows that the downstream company is under pressure to make the end product safe. But the downstream company may not have enough visibility into the model’s training data, capabilities, or failure modes to do that job effectively. The result is an end product that combines an undertested base model with a deployer that lacks the technical tools to compensate.

The prisoner’s dilemma at the heart of AI safety

The study frames this dynamic as a classic prisoner’s dilemma. In game theory, two rational actors must choose between cooperation and betrayal. If both cooperate, they achieve the best joint outcome. If one betrays while the other cooperates, the betrayer gains an advantage and the cooperator suffers. If both betray, they both end up worse off than they would have been through cooperation. Without a way to trust the other player, each tends to choose betrayal to protect their own interests.

In the AI supply chain, the two players are the general-purpose AI producer and the downstream domain specialist. Both need to invest in safety for the final product to be genuinely safe. But each knows that if the other invests a lot, they can invest a little and still receive some protection. If both decide to free-ride, safety collapses. Strict regulation can break this loop by requiring both sides to meet minimum standards, making cooperation the rational choice and enabling the best outcome for everyone.

The authors do not treat safety and commercial gain as inherently opposed. In their model, stronger, well-placed regulation can mutually benefit all players by improving both the safety of the end product and the utility each company gets from its investment. Utility, in the paper, is defined as revenue share minus investment cost. When regulators set clear expectations for all layers, companies can invest in safety without fearing that competitors will undercut them, and users get more trustworthy products.

How weak rules increase risk

Weak regulation is especially dangerous because it creates an illusion of oversight. Regulators, the public, and even corporate boards may believe that a problem has been addressed because a rule exists. But the rule may be written so narrowly that it misses the most consequential decisions in the AI lifecycle. If a company can claim compliance by testing only the final application, the underlying model may never be examined for dangerous capabilities, hidden biases, or vulnerabilities that emerge in unexpected contexts.

The paper’s insight is that this half-regulation can be worse than a legal vacuum. With no regulation, a downstream company that deploys an AI system knows it must take responsibility for the risks and is likely to demand more safety information from the model provider. With weak regulation, the downstream company may assume that the government has already vetted the technology, while the model provider assumes the same about the downstream company. In that gap, dangerous products can slip through.

The political and regulatory context

The study arrives at a moment of intense debate in Washington and Silicon Valley about how to govern AI. On one side, anti-regulation technologists argue that the United States needs minimal federal guardrails to move fast, innovate, and win the global AI race against China. They tend to view stricter rules as obstacles that create legal uncertainty and encourage companies to relocate research to friendlier jurisdictions. Some in this camp also accuse safety advocates of exaggerating risks to impose their own agenda.

On the other side, supporters of stricter federal regulation argue that the AI industry underestimates or undersells the dangers of under-regulated development. They point to concerns such as AI’s potential to cause economic disruption, the spread of misinformation, algorithmic bias, and the environmental and public health effects of energy-hungry data centers. The authors of the new study suggest that this debate misses an important point: regulation is not just a constraint on profit. It can create the conditions for both safer models and stronger commercial returns by encouraging cooperation throughout the supply chain.

What this means for policymakers

For lawmakers, the practical takeaway is that regulatory design matters as much as regulatory presence. A policy that appears pro-innovation because it imposes lighter burdens on AI developers may, in practice, weaken the entire ecosystem. The researchers say the most effective approach is to target the companies that train and develop AI models as well as the companies that adapt and deploy those models for specific use cases. This includes setting expectations for third-party audits, documentation, risk assessment, testing, incident reporting, and accountability at every stage of development.

Regulating the whole supply chain will not be simple. It requires new technical expertise at regulatory agencies, international coordination, and flexible rules that can adapt as models change. But the study suggests that the alternative — a fragmented patchwork of rules that only addresses the most visible layer of AI — may give the public a false sense of security while allowing the riskiest decisions to go unexamined.

The authors also note that AI is not a single object. It is a complicated network of stakeholders, each with different contributions, incentives, and levels of access to technical information. Laufer said that thoughtful regulation must therefore consider the whole supply chain, rather than focusing narrowly on one provider or one application. The model developers, the deployment companies, and the regulators all have a role to play. The question is how to design rules that encourage everyone to take safety seriously enough to make the final product trustworthy.

Source:Gizmodo News