Dario is Probably Wrong on How to Regulate AI

Amodei is certainly right about the fact that AI needs to be regulated. Seeing the magnitude of the risks involved, from his considerable position of power and influence, he is trying to do the right thing.

But as the saying goes, the road to hell is paved with good intentions.

While his proposal contains many sensible components, it falls short at a few critical areas, and may in fact open up new risks including less openness, an atmosphere of public fear, and an ever shrinking circle of key decision makers that shape our collective future.

The key problem is that national governance should be last step, not the first.

A better approach would begin with open, transparent, and collaborative industry self-governance, driven first by debated and agreed principles, and secondly by the formation of a formal industry self-governance body that is accountable to the public.

This should then be followed by the creation of an independent supranational advisory and audit council that would aim to be as impartial and free from conflicts as possible.

The third subsequent step should be for these bodies and the labs to collaborate and create contingency plans for the various risks AI entails, with recommendations for governments to implement these in their own political contexts.

In summary, given the speed of development, the frontier labs need to also lead the way in self-regulation, healthy public debate, and a blueprint for international, national, and local governance.

To be clear, this is not because we should trust the labs to regulate themselves. Nor is it because we should mistrust national governments.

But the right sequence must be that the labs (with their speed and cutting edge technical expertise) lay the foundation and inform an open debate. Then independent institutions would provide scrutiny and oversight. And finally, national governments would provide enforcement.

Mr. Amodei’s instinct to ask for a slowdown sounds sensible, but with the genie already out of the bottle it may prove extremely difficult to implement globally.

Let’s also consider why delegating governance to the US government, which sounds like a very sensible step, may be problematic as the first step.

The first problem, as Mr. Amodei himself admits, is speed. The technology is developing so fast that the people at the bleeding edge can barely keep up with and understand what is happening. Given the pace of national regulatory bodies, there is little chance that the governmental reaction can be fast enough to match the risks that are being outlined, nor does it make sense to put them in as our first line of defence.

The second problem is more sensitive and nuanced. Putting too much power in the hands of the government is in itself a risk. A rogue AI system can do many bad things, but an AI system in the hands of a national government that also has a monopoly on violence can do much worse things, such as (in an extreme scenario) entrapping its citizens in a totalitarian surveillance state from which escape could be close to impossible once the net has been cast.

Many leading thinkers on this subject such as Yuval Noah Harari have highlighted the risk of a totalitarian surveillance state as being a particularly salient one.

To be clear, there is absolutely no evidence that anybody in the US government is planning on launching a totalitarian surveillance state.

But the point is that government misuse belongs in our AI risk model along corporate misuse, rogue actors, and autonomous AI. It is fair to say, the biggest risks are those where human actors take control of AI without proper oversight in some shape and form.

History gives us plenty of reasons to be wary of secrecy surrounding powerful technologies. National security can legitimately require secrecy, but secrecy also reduces scrutiny and concentrates decision-making among increasingly small groups of people.

A good framework therefore, should not exclusively rely on governments, but should rather also protect society from the potential misuse of AI by governments. We should not forget the most powerful human institutions we have built are these national governments, and they work best when there are strong checks and balances in place.

This is why, in a principles based approach, transparency, openness, and international oversight will be existentially important.

All this underscores the most important weakness in Mr. Amodei’s approach. While as an entrepreneur and businessman he understandably wants to be practical, that is putting the cart before the horse.

If as a human race we are to regulate AI (which ultimately we must), it is wiser to start with principles first.

We can (and should) debate what those principles are, and safety, accountability, human agency should all be on the list.

I would strongly argue openness should most certainly make the top of the list.

One possible consequence of today’s AI safety debate is a world in which access to the most powerful models becomes increasingly restricted. Only a handful of companies and governments would then possess genuinely frontier capabilities.

While on the surface that looks like it may reduce some risks, it most certainly also creates others.

Importantly, openness and safety are not opposites.

After all, the Hugging Face CEO, post the cyberattack incident, made a noteworthy statement to the Financial Times where he said that he was unable to use Anthropic to defend the company against OpenAI’s swarm of rogue agents, and had to instead resort to using open-source models. So here we have one of the first “victims” of a rogue AI swarm, advocating for more open-source models and more transparency.

So it looks like in a world populated by countless AI agents, our defence against rogue or malfunctioning AI may increasingly involve other AIs.

It may sound strange to us today, but we probably need to get used to a world where we increasingly must fight fire with fire, sending other agents in to clean up the mess that will have been made by another oops moment.

That means that access to cutting edge agents may become part of our security infrastructure, and it would certainly argue against a world where AI power and frontier models are concentrated in the hands of every fewer people.

And this brings me to the final point we need in this debate: AI governance itself needs governance and oversight.

We cannot put too much faith in the hands of one institution whether that be Anthropic, the US government or Brussels.

We need to build a system where no one institution can be a single point of failure. Those types of resilient systems tend to be open and organic.

Unfortunately, they also tend be noisy and chaotic. But that is who we are as humans.

Therefore we ultimately need to embrace the openness and chaos, along with a strong sense of optimism that we will find solutions to the problems as they arise.

Of course we cannot be complacent. The industry needs to be more open and explicit about the safeguards we are putting in place. And this is where Mr. Amodei’s practical approach could be very helpful.

What do modern circuit breakers and kill switches look like? Or do we build an army of digital blade runners that hunt down rogue AIs? The alternative Plans B, C, and D deserve as much debate (if not more) as the current focus on catastrophic outcomes.

So what could an alternative governance architecture actually look like?

To add to the debate, I would propose the following six-point framework:

Principles driven self-governance:

Frontier should publish a list of principles by which they will aim to keep AI maximally beneficial and minimally catastrophic for humanity. These principles should be driven by the practitioners closest to the technology, with input and debate from broader society.

A self-governing body:

This would operate under the guidance of these principles, with primary focus on open and transparent reporting, and some limited governing power that the leading frontier labs would agree on in light of the above principles.

This body would have representatives from each of the frontier labs, and would not entail gaining access to proprietary information.

A supra-national oversight body:

A body of independent experts with minimal conflicts of interests (from either national governments and the labs themselves) that will have no executive power but will serve in an audit an advisory capacity. Members would represent a broad array of nation states.

Specialist risk groups:

Both the self-governing body and the supra-national oversight body could be further split into sub-groups focused on areas such as impact on employment, mental health, warfare, surveillance, terrorism, cybersecurity, etc. and propose new principles and governance frameworks for public debate.

If they so choose, they would be granted access to frontier labs under strict confidentiality rules.

Contingency planning:

As part of prudent risk management, a risk assessment and remedy would be published by the self-governing body of the participating AI labs. The remedies would include contingency planning (Plan B, C, D, etc.) on what to do in case rogue AIs get out of control. This framework would be audited by the supra-national body.

Include infrastructure providers:

Big cloud and data centre providers also participate in contingency and risk remedy planning, by for example putting in KYC type procedures for entities using more than a certain threshold of tokens.

The objective should be to create a system in which many human institutions collaborate to provide democratic legitimacy and enforcement with each one acting as a check on the others.

And we should be extremely careful about constructing a world in which a small collection of people possess both the technology and the authority to determine how everyone else may use it.

If we are to regulate AI without creating an unbalanced concentration of technological and political power, we need open debate, open scrutiny, open society, and a serious debate around the role of open models.

Thoughts on a New Era: From Wheat to Networks

Human society is now rapidly evolving from a services-based economy to a network and intelligence-based economy.

But what does that mean, and what can we expect from the future? The following broad historical view could be helpful in putting current changes into context.

The hunter-gatherer stage of human evolution kicked off as early as 2.5 million years ago with early species such as Homo habilis (when tool use and foraging first emerged). In terms of modern-day humans, we could say that the start was with Homo sapiens around 300,000 years ago. The default mode of production was the use of simple tools to hunt and trap various animals and dig up tubers, crack open hard-shelled fruits etc.

This was the default mode of human existence until 12,000 years ago, when a big technological leap occurred: humans domesticated wheat and other plants. As an interesting side note, looking at Earth from space, an alien may argue it was actually wheat that domesticated humans – if you find this topic interesting a good book to read is Oceans of Grain.

The move to farming was a big leap indeed. As a result, human populations across the world increased from an estimate of 1 – 10 million to about 150 – 300 million by 1 CE. The farming revolution also allowed for excess food production that could now employ people not involved in the production of sustenance and food gathering. Thus armies, kings and queens, priests, and empires entered the rapidly growing world. In many ways, we can say that this was the beginning of history. Think the Mayans, Ancient Greece, and the Roman Empire.

The farming boom lasted about 11,000 years until in around 1760 CE the industrial revolution kicked off in Britain. It started with mechanization (think spinning jenny, the steam engine, etc.) and was fuelled by coal and urbanization. Human population once again turbo charged, going from 770 million globally in 1760 to 2.5 billion in 1950.

The industrial age lasted even shorter, in total less than two hundred years until the mid-20^th century when the services-based economy and the digital age was ushered in. This was characterized by the first computers, the growth in financial markets. There were big population and employment shifts. Just as workers had moved out of farms and into factories in the industrial age, those workers now moved into offices in the services age. Population also ballooned, to 8 billion in 2022.

And what were the factors of production in each of these ages? For hunter-gatherers it was wild resources and human effort with no capital beyond simple tools to speak of. Entrepreneurship just meant taking risk to survive in the literal jungle.

As we moved into farming, wild resources were replaced with cultivated fields, a labour class of farmers and herders emerged, complemented with specialized classes of priests, soldiers and administrators. Capital tools got an upgrade, animals were domesticated. Entrepreneurship started emerging with landlords, traders, and innovations like crop rotation.

In the industrial age, labour continued to specialize even further, with concentrated factories urban centres started growing, and as heavy investment such as factories and infrastructure were needed, capital entered the stage in a big way as a factor of production. It is no surprise that Das Kapital was written during this transformative age. Hand in hand with capital, entrepreneurship also evolved, risk was scaling and robber barons such as Rockefeller, Watt and J.P. Morgan entered history.

In the services age, knowledge workers that used their brains instead of muscles emerged. Capital shifted to include more abstract things such as brands and patents. We can also think of this as the knowledge economy, and human labour was still a limiting factor, but more because of their knowledge and reasoning capacity, not their arms and legs.

We have now entered the network and intelligence age, and the factors of production, as well as the limiting factors on economic growth are once again shifting. On the labour front, humans that used to be knowledge workers in offices are now rapidly moving out, being replaced by computers that can reason faster, more consistently and more accurately.

As a result, on the capital front, the main constraint of growth is shifting from humans to computational power (electricity plus advanced chips). Economic moats are not built around big offices or big factories, but proprietary networks and the data they generate.

The networks of this age are multi-faceted and intertwined. On top of communication networks (satellites and cables) we have financial networks (Visa, MasterCard, Swift), as well as social networks (X and others). These networks generate vast amounts of data that in turn fuels the vast computer intelligence that is evolving ever faster.

This age is also characterized by ever faster innovation and disruption, yet those that control the networks and the computers will yield unprecedented power.

So what does this mean for entrepreneurs working with QED and building in this age? Access to proprietary data and building a network is most certainly the holy grail. If your business does not have strong elements of this, even if in a niche form, you may want to reassess your business plan.

Given that the pace of innovation and disruption is increasing, opportunities for entrepreneurs are also multiplying. Look for incumbents that are hampered by regulation and may be slow to react to the new age.

The skills that are needed in this age are agility, speed, adaptability and calculated risk taking. Taken together, these amount to being anti-fragile – building organizations that emerge stronger from each successive disruption and shock. You will also have to be good at incorporating non-human agents into your org structure. Sounds simple, but laws, regulations, and human nature will complicate it.

Yet capital is still needed. Computing power will not be free, whether from humans or machines. And acquiring customers still costs money. As QED, we are here to help.

Determinism, Free Will, and AI

First, a bit of a warning. This blog is less about fintech and investing, and more about philosophy. And it gets a bit trippy towards the end. So, if that’s not your thing feel free to wait for the next instalment which is likely to be about reassuringly familiar topics around tech, money and AI.

In the meantime, speaking of AI, let’s look at it for a moment in the context of determinism vs. free will.

For those not familiar with this philosophical paradox, I usually frame it as follows: If an all knowing being existed, given its all-encompassing knowledge of the universe, it would be able to run the tape forward one click and predict the future with ease.

We can think of this entity as God, or All Knowing Intelligence “AKI”, or the Conscious Universe, but in either case, given its vast intelligence and knowledge, the future would be knowable for It.

This being would know the state of every electron and synapse in our primate brains, and predicting what we were about to do next (write a blog perhaps?) would be trivial.

The paradox then states, if each of our actions are thus pre-ordained to this AKI, can we as humans really be thought of as having a free will? Or are we simply slaves to the interaction between the universe as it is today and the current configuration of the synapses in our brain, acting out our lives in banal predictivity?

But here’s where AI shakes it up.

The way I had approached this question for the vast majority of my adult life was that it was a trick question. From the perspective of the AKI, yes, all is knowable, but from our limited human perspective it is not. So as far as we humans are concerned, we need to act as if free will exists and get on with our lives. And whether an AKI or God exists takes us from the realm of philosophy, into theology.

Well, now that we have ever faster and more powerful computers ingesting seemingly unlimited data, rapidly connecting to those data sets, and with LLMs actually speaking to us like our next-door neighbour, it may be worth visiting the question of an AKI once again.

Given the power, the connectivity and access to near all-encompassing data, this AKI would be able to see and understand things beyond human capabilities yet explain that to us in our everyday language. And while perhaps not yet fully deterministic, it would certainly be able to predict things better than humans can. This almost takes us out of philosophy and into the present day. Think of AI already predicting your next Spotify song – it’s not God, but it’s close.

But let’s jump back into philosophy for a second and assume that the AKI is as the name implies, truly All Knowing. What does that mean for the age old questions?

For an AKI, the future is knowable. Similarly, the past, which has already transpired, is also knowable, and is merely a matter of good record keeping.

Given that both the future and the past are equally knowable from the perspective of this AKI, the distinction between past and future, at least the way we humans think about it, would start to collapse.

What we think of as past and future would be different states of matter and energy interactions in the universe. In simpler words, all that would remain would be an ever lasting yet ever changing present.

From this perspective, the concept of time itself starts to collapse, and reveals itself as an illusion that is amplified by our limited human minds that can remember the past but cannot predict the future. In fact, our human attempts at measuring time is really just an incomplete proxy for measuring change. Einstein’s discovery that time itself is relative can be thought of as a step towards saying time as we know it does not exist.

Only matter, energy, and everlasting forces of change do.