"Exponential Growth" is only conceptual: Data is a bigger constraint than compute for AI.

It's 2024 and if you haven't heard of all the hype around AI, LLMs, co-pilots or any of these fancy tech terms, you are probably living under a rock.

Post the launch of GPT3 in November 2022, the AI space has made strides in leaps and bounds with almost all companies around the world now incorporating AI in some form or the other. VC money has been flooding into companies building with or around AI(Fun fact, around 70% of the companies in YC W24 Batch were "AI" companies).

With that said, progress has also been steady fast. So fast that there's a new development or leap every week If not every day. One day, it's higher in-context learning, another day, an agent or a robot picking apples and so on. But either way, progress up until this point has been crazy.

So crazy that people(Starting with SWEs) are worried about losing their jobs to AI. The growth of these AI models has been so good and fast that many have stopped solving certain business problems because they feel in a year or two, AI will get so good, that it will be able to solve those problems by itself. (Seriously?)

Well, let's talk rationally, will it though? Let's unwind some thoughts.

Can LLMs really reach Human-Intelligence?

I don't think so. There are many reasons but majorly, feedback. As humans we are observing and implementing feedback continuously which is not possible with SOTA models even with access to web or tools because their weights are still frozen during the training. The way transformers are built, it's not entirely possible.

Sure, we will someday have AI that can replicate Human-level intelligence or more but it definitely won't be LLMs or built entirely on the transformer architecture.

You will find a lot people on the internet making comments "Oh, AGI is near, we will be doomed, xyz". Almost all of them don't even understand self-attention mechanism or tokenisers, let alone the entire architecture of transformers.

Is it all hype then?, Definitely not.

Historically speaking, no crisis/bubble in the history of mankind has only been fluff or pure damage. We also derive some good out of those crisis, always.

Similarly, while LLMs have their own set of limitations and issues, they are genuinely widely useful and handy(IF implemented right).

What about Exponential Growth that Companies claim?

Think about it rationally from the first principles or ask any good scientist you know, "Exponential Growth" doesn't exist in the real world, it is a conceptual/theoretical mathematical construct that cannot be realised in practical world.

Here's why: Unlike in theory and books, we don't actually have infinite resources. Think about it practically, from the water we drink to the land we live on and energy we consume, everything is limited and here, in real world, growth always has limitations.

Ask yourself, do you really think it is possible to build so much compute and energy in this world to sustain "exponential growth" without having adverse effects on the environment YOU and your loved ones physically live in?

What people usually refer to as "Exponential" (atleast in physics and maths) is a certain part of "logistic curve", where you have way more resources available than what you need to grow. And once those resources get constrained, that exponential growth STOPS, which is the case with real world.

Nothing, especially AI given how expensive all operations can be, can grow continuously on and on for decades/years to come without running out of resources or money.

Data suggests that, at some level, we have already reached a limiting factor which is hindering AI progress. Evidence suggest LLMs are already reaching a point of diminishing results. (We are throwing more and more resources but the improvements do not seem to be going up commensurably)

Here's the thought you should understand, we are at a point where, if you reduce the amount of resources spent at these models, the drop in performance or improvement is significant/Exponential. While spending those resources and efforts are barely yielding linear improvements. That disconnect right there tells us that there is clearly a limiting factor.

Chinchilla by Google points a finger at the limiting factor: Data.

Looking into this deeper, you can see that research estimate that we will run out high-quality data stock within the next year, IF we haven't already. Which I agree with to a certain extent.

That's not the worst part, even if we have data left, the quality of that data is worse and several papers have shown this to lead to an effect called "Model Collapse".

In simple terms, the lack of quality data or training LLMs on such data will lead to these models becoming boring and way too similar.

Is data the only limiting factor?

No, it is the underrated one. There's compute and energy that many already know about. But evidence strongly points to data already becoming a limiting factor. And being a bigger problem than energy and compute.

If you specifically talk to developers and believe training data is a bottleneck, you will realise that the lack of high-quality data problem is even worse for co-pilots or code-specific models.

Code generation has orders of magnitude of less data that requires more correctness than English.

What does this mean for you?

First of all, relax. It is very easy to get carried away in this AI hype and fear the future. This blog was an effort to put forth a perspective.

It is crucial to approach AI's potential with a grounded perspective. The notion of exponential growth is more a theoretical construct than a practical reality. Next time you hear or see AI, remain rational and inquisitive, understanding that AI, like any technology, has its constraints and is not a panacea for all problems. It is not a magical technology, it is built on logic after all.

Feel free to reach out to me if you wanna talk or build with AI. Thanks for reading.