How I’m Currently Thinking About AGI

There’s a lot of confusing information out there at the moment when it comes to the term ‘AGI’.

There’s also a lot of slop being posted by influencers with no idea what they are talking about, hyping everything up without nuance.

I would like to share a few things about my own beliefs of ‘AGI’ and how they’ve changed over the past few months in particular.

One of the biggest contentions at the moment in the field is this question of ‘well, if we simply scale up the existing models with more data and more compute, and maybe tweak some of the training methods pre and post, eventually we will get to AGI’.

I do not personally believe this, however, I do believe scaling the existing methods/models will be extraordinarily economically valuable, will automate large swaths of work (digital work in particular), and may in fact lead to AGI along the path by helping accelerate AI research breakthroughs that pertain to continual learning with inspiration from neuroscience.

There’s a lot to unpack with that statement, so let me elaborate.

First, what is AGI?

To me, artificial general intelligence is a system capable of continual learning in a general way. ‘General’ meaning it can be the best software engineer in the world, the best digital marketer in the world, the best tennis player in the world (if embodied in a robot with sufficient dexterity), etc. And it’s all the same system capable of elite (then superhuman) performance across all these domains.

However, it does not achieve this performance by training on all the code on the internet nor all the marketing data, nor watching/participating in all the tennis lessons available to it.

Rather, it achieves its performance through continual learning.

In other words, if we have the right architecture for AGI, it should be capable of starting at 0 and then gradually accumulating more and more information, correcting its representation of the accuracy of such information along the way, and storing it in an efficient, cost-effective, non-corruptible way. It’s capable of forming an accurate representation - a world model - of the environment we occupy (Earth) given the compute and tools available to it.

From this definition alone, scaling up the current AI models does not lead to AGI - without changes, it fails the definition because it is not sample efficient and does not continually learn at run time.

The current AI models are ‘pre-trained’ on effectively all data on the internet, and then in ‘post-training’ they are turned into chatbots (in the case of LLMs) via something called reinforcement learning from human feedback (RLHF).

You can think of this like: imagine a baby was born and internalised the representation of everything on the internet, but it was a one-time thing. The neural weights in their brain were then frozen at the age of 5, and some human expert came along and said

‘hey look, this is really weird, we have a baby with a specific type of dementia… they appear to be super smart and can regurgitate all these brilliant things about the world and they appear to even be creative in lots of ways, extrapolating insights that appear to be unique.

But there’s one problem - they can’t learn anything new. Then someone comes along and says ‘Hey, I know how we can make them learn new stuff, let’s get lots of examples of human experts asking questions and providing answers and then bias the model towards responding like those experts, and let’s give them access to tools humans also have access to, so they can retrieve and store new information (the internet, databases, etc)’.

Does that make the human example in this case a ‘general intelligence’?

Because that’s really what this whole debate comes down to.

In my opinion the answer is no. It makes it a different type of intelligence - something strange, interesting, extremely useful, but not like human intelligence because it cannot update its model of the world in real time - these models rely on the foundational model providers pushing releases to update them, like traditional software companies, and then giving them more and more context/tools to try make them more useful and human-like.

But they are not humans, and they are not general intelligences.

We can say they are intelligent, for sure, just not in the way humans are.

This is basically my mental model for how these AI models work. It’s not a totally accurate representation (obviously), but I’m using this example to point something specific out to you:

The pre-trained data these models train on, and even the reinforcement learning data, primarily relies on humans.

At its core, it’s human based.

But this is not how humans learn.

Babies learn to walk in all different ways, kind of like self-play. They have a goal and they figure out how to achieve the goal through error correction: try to walk, fail, try again, fail, train again, succeed (kind of), try again, succeed.

Humans continuously learn.

Which is why if you think of the smartest person you personally know, you could safely say that person could have chosen any career they would have liked. They could switch and within a few years be one of the best at it without really trying that hard.

Why?

Because they are sample efficient.

They can learn from fewer examples, and come up with richer, deeper representations of what each example actually means, then integrate it into their unified world model (i.e. how the world works).

AGI requires continual learning.

An artificial general intelligence has to be as sample efficient as humans, and has to be able to learn on the fly in an energy efficient way.

The only existence proof we have of this is the human brain.

The efficiency part of intelligence is not some random side thing.

It is the crux of what intelligence is.

I also find it interesting that people very frequently think that human memory (which is very fallible) is a bug rather than a feature.

I actually think it’s a feature.

Let me explain why.