The changing goalposts of AGI and timelines

We’ll know AGI when we see it, and this ain’t it. This complaining about changing goalposts is so transparently sour grapes from people over-invested in hyping the current LLM paradigm.

Thinking cool and all but not that extraordinary. Even plants does it.

I was thinking something similar - this isn't AI, and none of "those people" care if it is or isn't. They don't care philosophically, or even pragmatically.

They're selling a product. That product is the IDEA of replacement of the majority of human labor with what's basically slave labor but with substantially disregardable ethical quandaries.

It's honestly a genius product. I'm not surprised it's selling so well. I'm vaguely surprised so many people who don't stand to benefit in any way shape or form, or who will even potentially starve if it works out, are so keen on it. But there are always bootlickers.

The most unfortunate part is that when the party ends, it's none of "those people" who will suffer even in the slightest. I'm not even optimistic their egos will suffer, as Musk seems to show they are utterly immune even as their companies collapse under them.

"Highly autonomous systems" and "most economically valuable work" aren't precise enough to be useful.

"Highly" implies that there is a continuum, so where does directed end and autonomy begin?

"Most economically valuable work"... each word in that has wiggle room, not to mention that any reasonable interpretation of it is a shifting goalpost as the work done by humans over history has shifted a great deal.

The point is that none of this is defined in a way so that people can agree that something has AGI/ASI/etc. or not. If people can't agree then there's no point in talking about it.

EDIT: interestingly, the OpenAI definition of AGI specifically means that a subset of humans do not have AGI.

The definition reminds me of the common quip about robotics, "it's robotics when it doesn't work, once it works it's a machine".

If the definition has shifted once again to mean "a computer program that does a task pretty well for us", then what's the new term we're using to define human-level artificial intelligence?

Is doing a ton of heavy lifting. What is considered economically valuable work is going to change from decade to decade, if not from year to year. What’s considered economically valuable also is going to be way different depending across individuals and nations within the exact same time frames too.

And I get that there are workarounds; effectively a cron job every second prompting "do the next thing".

But in my personal definition of "highly autonomous" it would not need prompting at all. It would be thinking all the time, independently of requests.

what does "economically" means here? would it cover teaching? child care? healthcare? etc.

These days it is neural networks and transformer models for language in particular that people mean when they say unqualified AI.

It is very hard to have a meaningful discussion when different parties mean different things with the same words.

All humans fail to be AGI, by definition.

Yes.

The important context I think people may miss is, this does not require AI to be 10x or 5x or even 1x as good as a human programmer. Claude is worse than me in meaningful ways at the kind of code I need to write, but it’s still doing almost all my coding because after 4.6 it’s smart enough to understand when I explain what program it should have written.

'If you actually know what models are doing under the hood to product output that...'

Any one that tells you they know 'what models are dong under the hood' simply has no idea what they're talking about, and it's amazing how common this is.

Why this discussion is already annoying and poised to get so much worse is because now hundred billion dollar companies have a direct financial incentive to say they did it so I expect the definition will get softened to near meaninglessness so some marketing department can slap AGI on their thing.

Karpathy himself has publicly stated that AGI itself is only possible with a new paradigm (that his group is working toward). He claims RHLF and attention models are near the end of their logarithmic curve. The concept of the "self-training AI" is likely impossible without a new kind of model.

We will likely see some classes of human skills completely taken over by LLMs this decade: call centers (already capable in 2026), SWE (the next couple years). Bear in mind the frontier labs have spend many billions on exhaustive training on every aspect of these domains. They are focusing training on the highest value occupations, but the long tail is huge.

It will be interesting to see if this investment will be obviated by a "real AGI" capable of learning without going through the capital-intensive training steps of current models.

If we can crack long memory we're most of the way there. But you need RL in addition to long memory or the model doesn't improve. Part of the genius of humans is their adaptability. Show them how to make coffee with one coffee machine, they adapt to pretty much every other coffee machine; that's not just memory, that's RL. (Or a simpler example: crows are more capable of learning and acting with memory than an LLM is)

Currently the only way around both of these is brute-force (take in RL input from users/experiments, re-train the models constantly), and that's both very slow and error-prone (the flaws in models' thinking comes from lack of high-quality RL inputs). So without two major breakthoughs we're stuck tweaking what we got.

Sonnet 3.5 is old hat, and today's Sonnet 4.6 ships with an extra long 1M context window. And performs better on long context tasks while at it.

There are also attempts to address long context attention performance on the architectural side - streaming, learned KV dropout, differential attention. All of which can allow LLMs to sustain longer sessions and leverage longer contexts better.

If we're comparing to wet meat, then the closest thing humans have to context is working memory. Which humans also get a limited amount of - but can use to do complex work by loading things in and out of it. Which LLMs can also be trained to do. Today's tools like file search and context compression are crude versions of that.

The human eye processes between 100GB and 800GB of data per day. We then continuously learn and adapt from this firehose of information, using short-term and long-term memory, which is continuously retrained and weighted. This isn't "book knowledge", but the same capability is needed to continuously learn and reason on a human-equivalent level. You'd need a supercomputer to attempt it, for a single human's learning and reasoning.

RL is used for SOTA models, but it's a constant game of catch-up with limited data and processing. It's like self-driving cars. How many millions of miles have they already captured? Yet they still fail at some basic driving tasks. It's because the cars can't learn or form long-term memories, much less process and act on the vast amount of data a human can in real time. Same for LLM. Training and tweaking gets you pretty far, but not matching humans.

> With LoRA and friends it's also already possible to do continuous training that directly affects weights, it's just that economy of it is not that great

And that means we're stuck with non-AGI. Which is fine! We could've had flying cars decades ago, but that was hard, expensive and unnecessary, so we didn't do that. There's not enough money in the global economy to "spend" our way to AGI in a short timeframe, even if we wanted to spend it all, even if we could build all the datacenters quickly enough, which we can't (despite being a huge nation, there are many limitations).

> For some definitions of AGI

Changing the goalposts is dangerous. A lot of scary real-world stuff is hung on the idea of AGI being here or not. People will keep getting more and more freaked out and acting out if we're not clear on what is really happening. We don't have AGI. We have useful LLMs and VLMs.

If you've used the latest models extensively, you must've noticed times when AI 'runs out of common sense' and keeps trying stupid stuff.

I'm somewhat convinced that the amazing (and improving!) coding ability of these LLMs comes from it being RLHFd on the conversations its having with programmers, with each successfully resolved bug, implemented feature ending up in training data.

Thus we are involuntarily building the world's biggest stackoverflow.

Which for the record is incredibly useful, and may even put most programmers out of a job (who I think at that point should feel a bit stupid for letting this happen), but its not necessarily AGI.

Which fundamental limitation do you mean? I haven't seen anything but slow, iterative improvements. Sure, if feels fine, turtle can eventually do 10,000 mile trek but just because its moving left and right feet and decreasing the distance doesn't mean its getting there anytime soon.

Parent mentioned way harder hurdles than iterative increments can tackle, rather radical new... everything.

Humans generally do it on accident. They don't preface it with "Let me delete the production database," which LLMs do.

Humans do it accidentally.

The boundaries of these systems is very easy to find, though. Try to play any kind of game with them that isn't a prediction game, or perhaps even some that are (try to play chess with an LLM, it's amusing).

Eh? Which limitations were solved?

I am to sure one can really extrapolate much out of that, but I do find it interesting nonetheless.

I think language is also an important factor. I have a hard time deciding which of the two LLMs is worse at Swift, for example. They both seem equally great and awful in different ways.

This doesn't seem contradictory if you consider that success at AGI will solve the problem of carbon emissions, one way or another. If one data center ultimately replaces a whole medium-sized city of commuters...

I remember when computers became better than humans at chess, many people were shocked and saw that as machines becoming more intelligent than humans. Because being good at chess what considered equivalent to "being smart".

So... we can't tell when the rocket has left Earth atmosphere, but we can tell when the rocket has entered space?

I'm not getting how "superior in all tasks" is better-defined for you than "equal in all tasks".

I completely agree. We can't even measure each other well, let alone machines.

I guess what I failed to convey in my original comment was that, like the Internet 20 years ago, the current advancement made by AI might stall at a foundational level, while the landscape evolves.

Essentially, I believe what you're saying is really close in spirit to what I'm saying.

> It can be debated whether arena.ai is a suitable metric for AGI, a strong case can probably be made for why it’s not. However, that’s irrelevant, as the spirit of the self-sacrifice clause is to avoid an arms race, and we are clearly in one.

> Therefore, one can only conclude, that we currently meet the stated example triggering condition of “a better-than-even chance of success in the next two years”. As per its charter, OpenAI should stop competing with the likes of Anthropic and Gemini, and join forces, however that might look like.

The new title is a single, almost throwaway, line from the article.

> While this will never happen, I think it’s illustrative of some great points for pondering:

> The impotence of naive idealism in the face of economic incentives. The discrepancy between marketing points and practical actions. The changing goalposts of AGI and timelines. Notably, it’s common to now talk about ASI instead, implying we may have already achieved AGI, almost without noticing.

This does not work. From the guidelines:

> Please don't post on HN to ask or tell us something. Send it to hn@ycombinator.com.

https://news.ycombinator.com/newsguidelines.html

Is this really the garbage that should lead humanity to our future? Because inevitably that will be a dark future for 99.99% of the humans. And no, you and you won't be part of that 0.01% or whatever tiny number of elite think they are better than rest of us.

The point you're desperately trying to miss is that most other companies don't put up those moral claims in the first place

If big corporations do things that are unethical, they should be called out, even if they're common. Saying "well everyone's doing it", isn't a good excuse to do things that are unethical.

It's not "naive" to point out the lies that OpenAI told to get to the point that they are now. They were claiming to be a non-profit for awhile, they grew in popularity based in part on that early good-will, and now they are a for-profit company looking to IPO into one of the most valuable corporations on the planet. That's a weird thing. That's a thing that seems to be kind of antithetical to their initial purpose. People should point that out.

On its face that’s not a crazy stance: Governments are meant to represent the public, while private companies obviously aren't. I think it’s somewhat understandable why the government might reject that kind of "we know better than you" type of clause.

Of course, the reaction is wildly out of proportion. A normal response would just be to stop doing business with the company and move on. Labeling them a supply chain risk is an extreme response.

If you're one of the contractors working in NRO or aware of Sentient, OpenAI and Anthropic probably do look like supply chain risks. They want to subsume the work you're already doing with more extreme limitations (ones that might already be violated). So now you're pitching backup service providers, analyzing the cost of on-prem, and pricing out your own model training; it would be really convenient if OpenAI just agreed to terms. As a contractor, you can make them an offer so good that it would be career suicide to refuse it.

Autonomous weapons are a horse of a different color, but it's safe to assume the same discussions are happening inside Anduril et. al.

https://www.vp4association.com/aircraft-information-2/32-2/m...

A less charitable interpretation is that the current doctrine is "China / Russia will build autonomous killbots, so we can't allow a killbot gap".

I'm frankly less concerned about "proper" military uses than I am about the tech bleeding into the sphere of domestic law enforcement, as it inevitably will.

Hum...

The one thing domestic surveillance enables is defining targets inside the country, and the one thing lethal autonomy enables is executing targets that a soldier would refuse to.

Those things don't have other uses.

A 2017 national intelligence law compels Chinese companies and individuals to cooperate with state intelligence when asked and without and public notice.

China has no equivalent of the whistleblower protection that enables resignations with public letters explaining why, protests, open letters with many signatures, etc. Whenever you see "Chinese whistleblower" in the news, you're looking at someone who quietly fled the country first and then blew the whistle. Example: https://www.cnn.com/2026/02/27/us/china-nyc-whistleblower-uf...

Not to mention that the principles are not being betrayed now for the first time.

since when did the view that "humans should be in the loop before murderbots target and kill someone" become a "naive moral absolutist view of the world"? we're resigned to building the terminator now?

There are a number of industries where I don’t think “kinda…” is an acceptable answer to “was this code read before deploying?”. Humans aren’t great at repeating boring tasks ad nauseam.

I don’t even think humans can “move beyond” their sensory data. They generalize using it, which is amazing, but they are still limited by it.* So why is this a reasonable standard for non-biological intelligence?

We have compelling evidence that both can learn in unsupervised settings. (I grant one has to wrap a transformer model with a training harness, but how can anyone sincerely consider this as a disqualifier while admitting that an infant cannot raise itself from birth!)

I’m happy to discuss nuance like different architectures (carbon versus silicon, neurons versus ANNs, etc), but the human tendency to move the goalposts is not something to be proud of. We really need to stop doing this.

* Jeff Hawkins describes the brain as relentlessly searching for invariants from its sensory data. It finds patterns in them and generalizes.