Amodei is certainly right about the fact that AI needs to be regulated. Seeing the magnitude of the risks involved, from his considerable position of power and influence, he is trying to do the right thing.
But as the saying goes, the road to hell is paved with good intentions.
While his proposal contains many sensible components, it falls short at a few critical areas, and may in fact open up new risks including less openness, an atmosphere of public fear, and an ever shrinking circle of key decision makers that shape our collective future.
The key problem is that national governance should be last step, not the first.
A better approach would begin with open, transparent, and collaborative industry self-governance, driven first by debated and agreed principles, and secondly by the formation of a formal industry self-governance body that is accountable to the public.
This should then be followed by the creation of an independent supranational advisory and audit council that would aim to be as impartial and free from conflicts as possible.
The third subsequent step should be for these bodies and the labs to collaborate and create contingency plans for the various risks AI entails, with recommendations for governments to implement these in their own political contexts.
In summary, given the speed of development, the frontier labs need to also lead the way in self-regulation, healthy public debate, and a blueprint for international, national, and local governance.
To be clear, this is not because we should trust the labs to regulate themselves. Nor is it because we should mistrust national governments.
But the right sequence must be that the labs (with their speed and cutting edge technical expertise) lay the foundation and inform an open debate. Then independent institutions would provide scrutiny and oversight. And finally, national governments would provide enforcement.
Mr. Amodei’s instinct to ask for a slowdown sounds sensible, but with the genie already out of the bottle it may prove extremely difficult to implement globally.
Let’s also consider why delegating governance to the US government, which sounds like a very sensible step, may be problematic as the first step.
The first problem, as Mr. Amodei himself admits, is speed. The technology is developing so fast that the people at the bleeding edge can barely keep up with and understand what is happening. Given the pace of national regulatory bodies, there is little chance that the governmental reaction can be fast enough to match the risks that are being outlined, nor does it make sense to put them in as our first line of defence.
The second problem is more sensitive and nuanced. Putting too much power in the hands of the government is in itself a risk. A rogue AI system can do many bad things, but an AI system in the hands of a national government that also has a monopoly on violence can do much worse things, such as (in an extreme scenario) entrapping its citizens in a totalitarian surveillance state from which escape could be close to impossible once the net has been cast.
Many leading thinkers on this subject such as Yuval Noah Harari have highlighted the risk of a totalitarian surveillance state as being a particularly salient one.
To be clear, there is absolutely no evidence that anybody in the US government is planning on launching a totalitarian surveillance state.
But the point is that government misuse belongs in our AI risk model along corporate misuse, rogue actors, and autonomous AI. It is fair to say, the biggest risks are those where human actors take control of AI without proper oversight in some shape and form.
History gives us plenty of reasons to be wary of secrecy surrounding powerful technologies. National security can legitimately require secrecy, but secrecy also reduces scrutiny and concentrates decision-making among increasingly small groups of people.
A good framework therefore, should not exclusively rely on governments, but should rather also protect society from the potential misuse of AI by governments. We should not forget the most powerful human institutions we have built are these national governments, and they work best when there are strong checks and balances in place.
This is why, in a principles based approach, transparency, openness, and international oversight will be existentially important.
All this underscores the most important weakness in Mr. Amodei’s approach. While as an entrepreneur and businessman he understandably wants to be practical, that is putting the cart before the horse.
If as a human race we are to regulate AI (which ultimately we must), it is wiser to start with principles first.
We can (and should) debate what those principles are, and safety, accountability, human agency should all be on the list.
I would strongly argue openness should most certainly make the top of the list.
One possible consequence of today’s AI safety debate is a world in which access to the most powerful models becomes increasingly restricted. Only a handful of companies and governments would then possess genuinely frontier capabilities.
While on the surface that looks like it may reduce some risks, it most certainly also creates others.
Importantly, openness and safety are not opposites.
After all, the Hugging Face CEO, post the cyberattack incident, made a noteworthy statement to the Financial Times where he said that he was unable to use Anthropic to defend the company against OpenAI’s swarm of rogue agents, and had to instead resort to using open-source models. So here we have one of the first “victims” of a rogue AI swarm, advocating for more open-source models and more transparency.
So it looks like in a world populated by countless AI agents, our defence against rogue or malfunctioning AI may increasingly involve other AIs.
It may sound strange to us today, but we probably need to get used to a world where we increasingly must fight fire with fire, sending other agents in to clean up the mess that will have been made by another oops moment.
That means that access to cutting edge agents may become part of our security infrastructure, and it would certainly argue against a world where AI power and frontier models are concentrated in the hands of every fewer people.
And this brings me to the final point we need in this debate: AI governance itself needs governance and oversight.
We cannot put too much faith in the hands of one institution whether that be Anthropic, the US government or Brussels.
We need to build a system where no one institution can be a single point of failure. Those types of resilient systems tend to be open and organic.
Unfortunately, they also tend be noisy and chaotic. But that is who we are as humans.
Therefore we ultimately need to embrace the openness and chaos, along with a strong sense of optimism that we will find solutions to the problems as they arise.
Of course we cannot be complacent. The industry needs to be more open and explicit about the safeguards we are putting in place. And this is where Mr. Amodei’s practical approach could be very helpful.
What do modern circuit breakers and kill switches look like? Or do we build an army of digital blade runners that hunt down rogue AIs? The alternative Plans B, C, and D deserve as much debate (if not more) as the current focus on catastrophic outcomes.
So what could an alternative governance architecture actually look like?
To add to the debate, I would propose the following six-point framework:
- Principles driven self-governance:
Frontier should publish a list of principles by which they will aim to keep AI maximally beneficial and minimally catastrophic for humanity. These principles should be driven by the practitioners closest to the technology, with input and debate from broader society.
- A self-governing body:
This would operate under the guidance of these principles, with primary focus on open and transparent reporting, and some limited governing power that the leading frontier labs would agree on in light of the above principles.
This body would have representatives from each of the frontier labs, and would not entail gaining access to proprietary information.
- A supra-national oversight body:
A body of independent experts with minimal conflicts of interests (from either national governments and the labs themselves) that will have no executive power but will serve in an audit an advisory capacity. Members would represent a broad array of nation states.
- Specialist risk groups:
Both the self-governing body and the supra-national oversight body could be further split into sub-groups focused on areas such as impact on employment, mental health, warfare, surveillance, terrorism, cybersecurity, etc. and propose new principles and governance frameworks for public debate.
If they so choose, they would be granted access to frontier labs under strict confidentiality rules.
- Contingency planning:
As part of prudent risk management, a risk assessment and remedy would be published by the self-governing body of the participating AI labs. The remedies would include contingency planning (Plan B, C, D, etc.) on what to do in case rogue AIs get out of control. This framework would be audited by the supra-national body.
- Include infrastructure providers:
Big cloud and data centre providers also participate in contingency and risk remedy planning, by for example putting in KYC type procedures for entities using more than a certain threshold of tokens.
The objective should be to create a system in which many human institutions collaborate to provide democratic legitimacy and enforcement with each one acting as a check on the others.
And we should be extremely careful about constructing a world in which a small collection of people possess both the technology and the authority to determine how everyone else may use it.
If we are to regulate AI without creating an unbalanced concentration of technological and political power, we need open debate, open scrutiny, open society, and a serious debate around the role of open models.
Yusuf's blog 