Executive Summary

Background

Auto-Regressive Large Language Models (AR LLMs) is an emerging technology gaining increasingly broad real-world use. As these models become larger and larger their capacity and open-endedness create a difficult-to-predict scope of emergent behavior. Developers of AR LLMs regularly discover unexpected model capabilities.

Which stage of the Gartner hype cycle is “people lose their fucking minds and argue that we should start a nuclear war to prevent people from using maths”? Asking for a friend.

-- Benedict Evans ( @benedictevans) March 30, 2023

Due to the enigmatic internals of these models, it is challenging to make predictions of future capabilities using the scientific method. Central to the scientific method is falsifiability, i.e., that empirical tests can logically contradict theories or hypotheses. Currently, the scientific method is struggling to keep up with engineering progress. Engineers are making much faster progress on scaling up models, resulting in emergent behavior researchers cannot fully explain. This presents a problem for everyone observing progress in AI: where are we heading? Without accepted scientific theories, people start extrapolating based on their interpretation. As a result, people who are usually sensible can suddenly end up with extreme views. They also tend to oscillate quickly between being impressed and scared. This post attempts to make sense of the different theories and develop a position on the question of “What does the future of AI in society look like?”.

First of all: I’ve always taken the position that existing artificial intelligence is far worse than human intelligence. I started learning about machine learning in the late 1990s, and every time I learn about the inner workings of a new machine-learning algorithm, I feel, “Is this it?”. It is hard to feel that something is intelligent when you understand the mechanics. Even Transformers evoke this feeling for me. The algorithm is short and simple and not something that feels nearly as sophisticated and complex as the human brain. Just read a few pages from any of the great books Nick Lane has written (e.g., The Ten Great Inventions of Evolution), and you’ll realize how much goes into our biological system. Coming into the question of AGI, my starting point is: we’re far from human-like intelligence, and we should be humble.

What trajectories do people imagine when forecasting the future of AI?

To grossly oversimplify, there appears to be a matrix with two dimensions:

I’ll focus on three groups that I’ll call: “The Optimists,” “The Pessimists,” and “AGI is Distant.”

The Optimists. You can tell a plausible story where humanity greatly benefits from the emergence of AGI. Assuming we can reign in AGI and use it for the betterment of humanity, an intelligence vastly more capable than a human could unlock secrets of the universe we can only dream of today.

The Pessimists. You can also tell a plausible story in which AGI has a very negative impact on humanity. AI can be much more intelligent than humans and use information more efficiently. For example, AlphaZero learned to be superhuman at Go in only a few days. A misaligned AI smarter than humans could cause human extinction. It would also be virtually impossible to stop it if things go off the rails.

I think that the magnitude of the AI alignment problem has been ridiculously overblown & our ability to solve it widely underestimated.

I've been publicly called stupid before, but never as often as by the "AI is a significant existential risk" crowd.

That's OK, I'm used to it.

-- Yann LeCun (@ylecun) March 20, 2023

AGI is Distant. You could also argue that current systems are far from comparable to humans, let alone super-intelligent. The fact that super-intelligence is so distant makes the debate about the impact of AGI hypothetical to the point of being a waste of time. Yann LeCun is one of the most experienced and knowledgeable people who has taken this position. He outlines his position in this deck (e.g., page 9: “Unpopular Opinion about AR-LLMs: AR LLMs are doomed.”)

AGI Timeline

How likely is it that we will see super-intelligent algorithms in, say, the next five years? Most people reason about this in the context of the so-called Scaling Hypothesis, i.e., that models get better the larger they get. Google BIG-bench has an excellent overview of progress so far that I keep referring to:

Since models are getting larger and larger, and access to training data keeps increasing, you could make the case that we are in the early stages of an exponential trajectory. There is both quantitative and qualitative evidence of improved performance, so there is no denying this trend exists. But you could also argue that performance evaluations are limited in scope and that we will find performance on broader, real-world tasks less impressive.

Since we cannot explain the connection between the number of parameters and the reasoning capabilities of transformer-based models, projections are speculation. No one knows if we will face diminishing returns or not. Some forms of progress arrive in an S-curve-like way. Early progress is slow; then, there is a rapid rush of improvement, followed by a plateau of diminishing returns. This could very well prove to be true for current models.

Without an agreed-upon way to predict and explain how capabilities are related to the internal mechanics of these models, the scientific method is left in the dust cloud left behind by engineers. Whenever important issues become a matter of speculation, things quickly get messy. People project their values, preferences, and ideology on the issue, complicating public discourse. Add to that the drastic impact that self-improvement could add. Those who argue that super-intelligent systems could arrive very soon speculate that Auto-Regressive Large Language Models might be capable of self-improvement, unleashing a rapid and uncontrollable force.

My position on “AGI Timeline”: Empirical evidence indicates we will see continued performance improvements from auto-regressive language models, but there are no guarantees we won’t start seeing diminishing returns. I think models will keep getting better on many forms of tasks, but some tasks will be out of reach for this paradigm of models (e.g., searching decision trees, optimizing policies, and symbolic reasoning). Furthermore, I find it unlikely (<5%) that current auto-regressive language models become capable of self-improvement, but I cannot altogether reject the possibility of it happening. I have, however, started feeling comfortable calling LLMs a form of AI and not just mere machine-learning models. And I consider AGI -like systems to be already available today.

AGI Impact

I propose the Yann-Jaan continuum of existential risk from a misaligned AGI. At one end you have @ylecun who argues we are a long way off AGI and that alignment will be easy. At the other end you have Jaan Tallinn, signatory to the moratorium letter.

-- Ian Hogarth ( @soundboy) April 4, 2023

This is the most speculative aspect. The shortest version of this section would be: we have no idea how the existence of artificial super-intelligence will impact humanity since it doesn’t exist, and we have no experience on which to base our forecast. If you think that AGI is imminent, then by all means, speculate. I’m open-minded and think many versions of life with AGI are plausible, ranging from amazing to catastrophic.

My position on “AGI Impact”: I embrace the benefits of advanced machine learning and nascent AI as predominantly positive. We will be able to get rid of tons of tedious work and unlock new, amazing products that will improve the quality of life for humans. We will also face new challenges, such as rampant misinformation, carbon emissions, etc. This will require significant effort to handle, but it can be done. At the same time, I cannot altogether reject the risks that super-intelligence comes with, simply because there’s very little based on which we can predict impact. In the financial world, this would be referred to as tail risk: it’s unlikely that AGI is an existential threat to humanity, but it cannot be ruled out, and if it happens, the impact is catastrophic. If and how you hedge against tail risk is a matter of preference.

Practicalities

Leaving the issue of AGI aside, there is a long list of short-term challenges we must work on to make AI a net positive for humanity. This list includes issues like:

While the negative impact of each of these issues is arguably less catastrophic than a super-intelligence causing the extinction of humanity, they are also significantly more likely problems.

In closing: I choose to be an optimist, and I think AI will improve the quality of life for humans. We should enjoy all the great products we can build now while dealing with the practical issues we know to be real. Even if I cannot reject the possibility that there is a higher-order bit we might be getting wrong.