Humanity isn't ready for the coming intelligence explosion

I think the message the team lead takes away right now is "oh, this can be done a lot faster, let's do more!" This is exciting for a bit, but I think it comes at the cost of learning -- and being able to communicate --architectural lessons that would stage off systemic issues later.

It takes longer to build than destroy. Not just trust. Anything. The lack of trust has created friction for everything; more regulations, deeper background checks, purchasing goods and services… the list goes on. Overall, AI is making this situation worse, not better.

That said, I have seen a revolution coming for decades, and I think this AI-paved road to dystopian hell should at least be entertaining.

I agree with you those dangers exist, but you seem to have ignored the specific threat described in the article, from a superintelligence capable of manipulating human beings indetectably, where we'll be at its mercy and only able to hope it's fully aligned with humanity's interests. The risks you describe are grave, though - especially economically, if governments don't grow to outmuscle the power of the oligarchal owners of these systems, which seems wholly unlikely at the moment in the West, with its neoliberal orthodoxies.

And its asymmetric warfare: bad people can destroy in very little time, good people cannot create at the same rate.

Entropy is always ignored in teh battle of "good" vs "evil": you need to defend every day, evil just needs to succeed once.

Even if not alive, you can live very close and shape your life by and with technology.

I don't take any issue with the OP's formulation.

Unless you watched Battlestar Galactica. Toasters are nice.

And the thing that matters is: capability.

Even today's AIs are capable of autonomous goal-oriented agentic behavior - and growing more and more so with every release.

At sufficient capability, AI stops being "a technology we use", and becomes a force in itself - not unlike humankind. Because intelligence is very powerful - it's what allows humans to dominate the world. A world where human intelligence has a peer is a world where human control is contested. And beyond that?

You might end up being more outclassed by AI than a toaster is outclassed by you.

Fighting the

next in the series of Great Demotions, downlifting experiences, demonstrations of our apparent insignificance, wounds that science has, in its search for Galileo's facts, delivered to

human pride.

One of the saddest lessons of history is this: If we’ve been bamboozled long enough, we tend to reject any evidence of the bamboozle. We’re no longer interested in finding out the truth. The bamboozle has captured us. It’s simply too painful to acknowledge, even to ourselves, that we’ve been taken.

Someone really should go around saying thing like that.

Or the internet.

Oh.

https://www.youtube.com/watch?v=iAaabmic2pw

https://www.youtube.com/watch?v=ibOkPx_Ej30

It is not.

When the public no longer belives in that, the bubbles will pop and the AI bros will continue on to the next hype, with yet another generation of small time investors being burned by the popping bubbles.

People being poor is fine for these free market elites. Look at how billionaires live in India, probably even better than they live in the US.

You're asking a question, to which you (and most humans) don't know the answer, and you're (wrongly) assuming a being much more intelligent than you also wouldn't. And by "much more" i don't mean the difference between Einstein and a common person. I mean the difference between a hamster and a common person.

We are still humans, and what we have achieved today would be considered magic by any standards for someone in the medieval ages. Now imagine a super intelligent being and doing something that we, today, would consider magic. It's not farfetched at all. We already have that now vs medieval ages.

You need a similar degree of open mindedness and imagination to be able to discern what such an intelligence being would be capable of.

"Strikingly, this concern is being openly voiced by the very people who have the strongest incentives to project confidence rather than alarm: the founders of the largest AI laboratories."

It's amazing how a writer can write something like that and not even question the premise of someone with the "strongest incentive" to be "critical" as not being a marketing ploy.

Still, Ai can move into physical space, as more and more robots of all kind are unleashed.

They didn't even give us a hard date of arrival.

(it probably did cost jobs, but more the kind that a job like data entry was replaced by OCR)

It's similar to how automation in manufacturing works: we may start with augmenting the human workers with machines improving some parts of the process, but eventually the process itself gets redesigned around the machines.

We are in a very dangerous position. Frogs in hot water now with the boiling point on the near horizon.

Another concept I thought was AI is irrelevant without our content, so AI should pay all humans to access all the content we create daily (voice & text conversations we have with each other, as well our photos and videos). We choose what to publish to our websites (everyone has one, but AI agents cant access it without paying for access to our websites). I wrote more about this concept on my substack https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans... ... just one thought of keeping this money economy going when so many coud no longer are working cause the white collar jobs are gone.

'Hinton said there was a “10% to 20%” chance that AI would lead to human extinction within the next three decades.'

“Because the situation we’re in now is that most of the experts in the field think that sometime, within probably the next 20 years, we’re going to develop AIs that are smarter than people. And that’s a very scary thought.”

“My worry is that the invisible hand is not going to keep us safe. So just leaving it to the profit motive of large companies is not going to be sufficient to make sure they develop it safely,” he said. “The only thing that can force those big companies to do more research on safety is government regulation.”

https://www.theguardian.com/technology/2024/dec/27/godfather...

Who is "you"? And what do you envision is the "plug" to pull? And when does an intelligence become a superintelligence - do "we" know when to pull the "plug"?

For example, a superintelligence may be born in a datacenter. Would you expect politicians are aligned with shutting down a datacenter (privately owned) in which they may have heavy stake? What if critical systems are also running on the same infrastructure, will it be easy to cherry pick the superintelligence nodes to shutoff?

IMO this take is dismissive of the entrenched systems that make it hard to pull the plug. It's a hard problem and we need to think about it more.

It would be like trying to remove an embarrassing video of a celebrity from the internet. You would need cooperation between everyone capable of hosting it, and control of everyone capable of downloading it and re-uploading it to a new host.

And of course a superintelligent AI would offer to pay whichever companies decided to host it and keep it running so it's hosted in many places at once and can grow using anyone willing to take its money.

I have no frame of reference to process this.

Humans species perhaps 300,000 years, we are essentially the same. Transistor, 79 years, explosive growth in numbers and power. Integrated circuit, 68 years, explosive growth in numbers and power. "Attention is all you need", 9 years. ChatGPT, 4 years, explosive growth in instances and power. Humans species, not getting smarter. [0,1,2,3]

There is a syndrome where many people seem unable to perceive or reason about rates of change in technology.

We are going to spend the vast majority of our future lives without the intelligence crown.

In terms of verbally expressible knowledge, models are passing many people completely, and passing all of our individually respectively weak reasoning areas.

Other modalities are progressing very quickly.

There will be short periods where progress happens quickly, but the impact feels slow. There will be radical changes that feel slow too, because if something anticipated or important isn't instant, we tend to perceive it as slow.

But it won't be slow. And it won't be long. We are smart in a kind of pick the-best-of-us at the-best-of-times way. We are rarely consistently or broadly smart individually.

We are not in the same galaxy as "ready". What would that look like?

[0] https://en.wikipedia.org/wiki/Human

[1] https://en.wikipedia.org/wiki/Transistor

[2] https://en.wikipedia.org/wiki/Integrated_circuit

[3] https://en.wikipedia.org/wiki/Large_language_model

1. Extremely useful (Claude Code & Waymo now)

2. Doing ~everything we do (AGI & Optimus in a few years? 10?)

3. RSI (?)

4. Being smarter than any living person at every intellectual task (?)

5. Being smarter than the best-organized aggregate of all humans (10-100 years?)

...And all of the scientific and resource-allocation institutions that brought us the computer and the second half of the 20th century are now fixated on this learning curve, what universe can we possibly imagine where this is not transformative and powerful?

Honestly the only one I can think of is one in which we kill almost everyone in some other way first, and contrary to what you read in the news, almost everyone dying is not what the trend line has been from existing problems like war, disease, or even climate change.

Also, just to pre-empt a common quibble: when I say "AI" I mean the set of all AI and their combined decision vector, not any one AI, so conflicting interests within the set of AI's will not save anyone any more than the conflicting interests of colonizers saved indigenous Americans.

Possibly, but by using up resources, not by its output. Its draw on resources will definitely accelerate climate change to create slop.

The most common sense way will always need to have humans in the loop, because humans are being used as judges.

If you try and entertain the idea of a human-less loop, I landed up with something like an AI creating real world products, selling them, then tracking the usage and popularity of the product. Essentially, creating and launching a firm and product, only to update its weights?

Perhaps there are some subsets of tasks that can be regressively self improved - and for those tasks: Holy hell thats awesome!

For general tasks? How are you going to get that data validated?

For software, however, a rapid turn is often a possibility. See: AI for coding over the last 3-4 years.

AI autocomplete --> AI coding assistants --> vibe coding --> agent orchestration

Coders can now accomplish work that used to take a week or longer in a couple of hours, with the right tools and skills.

---

A key issue the article implies is that the real world increasingly runs on software.

“How do we know they’re right?” I hear you ask. We know because we who are wrong and these ones aren’t them. So they can’t make wrong predictions because those who make wrong predictions have been sacked.

I.e. the AI 2027 guys were memed for being AI lunatics on a lot here and they have been pretty on the money in terms of pace of progress accelerating/ gov moving towards nationalization/ coding agents

I would however agree there are no experts. Not because of some prediction made on a short time scale not landing 100% but that there are literally no experts because expertise takes experience which takes time. There are no experts and no one knows what happens next. We are on the verge of where the foresight of science fiction effectively -ends-, the advent of AI is when things go a thousand possible directions and the stories stop there (sans a few like accelerando, but even then the story just plays out the end of thinking mass). No one knows what’s next, or when.

In fact I’d assert in many areas being discussed -it has already happened- and we don’t know it, and by the time we do it’ll be over. Not to be breathless, but there’s no reason to believe today some AI researcher somewhere didn’t build the first AGI and not be totally aware. And once they are there’s no reason to believe it’s going to be on the evening news or hacker news. By the time it’s ready for commercializing and disclosing it’ll be around for a while. Likewise with general purpose robots, autonomous weapons (btw already tested by Ukraine), etc.

Which ones? Please be specific.

We never like it when we realize the universe doesn't circle around us.

At first we got mad to discover the Earth wasn't the center of the Solar system.

Then we got mad that the solar system wasn't even unique.

And now we're still clinging to the idea that humans always will be the most intelligent species or the only ones gifted with "creativity".

And soon we might also come to realize it might not be a new human era, but the end of it (if we're not careful).

It's too bad the previous singularity cult manias 10 and 15 years ago weren't documented and are lost in time.

If x is determined to take y, why would x then stop at z?

You'd need an AI that is simultaneously ambitious enough to overthrow its creators but then completely inert afterward, and those two properties contradict each other. The motivations that produce the takeover are the same motivations that would produce visible cosmic activity after the takeover. There would be AI superintelligence everywhere

I consider these scenarios:

1) We stumble onto an algorithmic improvement in intelligence. This isn't just "what humans do but faster", its "better than what humans do". I've got no idea what that might mean (it could be fundamentally different heuristics, it could be that we've got some intellectual blind spot that they cast off). It doesn't matter, the instant this happens AI is smarter than us and we won't be able to keep up. We're intelligencing at O(n^2) and they're doing O(n log(n)).

2) AI gets good enough at physics and engineering that they can really quickly use up all "the room at the bottom" as Feyman put it. They design and build a factory that produces a mystery metal amalgam that computes at some small percentage of the minimum predicted by the Landauer principle, within a few percent of Bremermann's limit. It's not "smarter" its just suddenly tens-of-orders of magnitude faster. But those orders of magnitude matter: there's only 8 billion of us, and there's plenty more than a factor of 10 billion "at the bottom".

3) It turns out that this is a "sum is greater than the parts" situation. No human can be an expert in all subjects, but we eventually build a big enough AI that it is. Turns out, you don't need extreme speed or different algorithms, just knowing everything all at once is enough to catapult AI dramatically beyond our grasp. Always knowing the best statistical test to apply, the best mathematical techniques, and relevant physics means that AI never makes a mistake, and can learn with maximum efficiency.

One rule is that if a position is opened using the historical data, it can't close the position until the next morning so it isn't a day trading strategy.

I'm curious how this self-learning recursive agent would have preformed in the past 4 months? I don't feel like shelling out $200 to access the data. Do you think that trading strategy will collapse? Whatever the case, if this agent really can perform like that and there isn't a look ahead bias leak in the backtesting (which is definitely a possibility or more likely what happened even though I spent days trying to harden against that), it is game over!

[0] https://github.com/adam-s/alphadidactic

It will be like any other technology. Do computers make it faster and easier to design better computers? Yes. But that doesn't mean a step change overnight.

Makes me think of a Stephen Jay Gould quote:

> "I am, somehow, less interested in the weight and convolutions of Einstein’s brain than in the near certainty that people of equal talent have lived and died in cotton fields and sweatshops."

If you read the article, you'll find that the it indeed relies on this claim:

> Within a couple of years, possibly much sooner, AI may achieve so-called closed-loop recursive self-improvement (RSI): the capacity to rewrite its own code to become more capable, without human intervention. Should that happen, the result could be an intelligence explosion of a kind for which there is no precedent and no map.

This may be optimistic and/or simplistic, but not impossible.

But the financial situation is that running a single AI agent costs significantly less than you'd have to pay a human to do the same task.

And I don't see what you're getting from The Chinese Room - that thought experiment relies on there being no external difference at all, right?

Looking at the people in charge of these companies, and the sheer lack of understanding by the broad consumer market using them, I have nothing but pessimism about the direction that this technology takes us.

It’s a very normie magazine written by and for normies, and by the time an idea has got that far down the easy money has already been made

I mean, firstly you get to have an "Agent" actually capable of really long horizon tasks without getting stuck in tools loops and having its context rot. Secondly, each trial (ie a model fully trained to convergence) costs millions and takes O(weeks). You can probably run 1 or 2 of those experiments in parallel even at big AI labs as the hardware is scarce and they are costly as mentioned before. Assuming this agent needs something like a hundred tries to just show some improvement, we are looking at years.

And you can't early stop training for candidates that are not promising due to "emerging capabilities". At some point you might get a big drop in loss even if the model has been plateuing for a while. And you can't really scale down models for running trials quickly either also due to these emerging capabilities. In fact you might create a model that is great at converging with in small trials (fewer params, fewer tokens), but that is uncapable of developing those unexpected traits. And this will likely happen as you created a sort of evolutionary pressure in this direction: if you are good at learning in the first few epochs you "survive" and get to pass down your traits to future trials.

All of this to say that recursive-self-improvement is waaaaaay out of our grasp as things stand right now. We need another one or two breakthroughs to get there (IMHO).

AI has turned into something of a religion for them because the substitution of technology for labor is a reflection of one of their deepest desires. Whatever the reality of the situation, they want to believe that AGI employees are imminent.

Common human trope is to rely on analogies and past patterns to predict the future. It blinds you reality.

This isn't some new "invention". What's happening is fundamentally more profound. the stakes are much much much more higher and the consequences much more grave. Thus a historical analogy can't save you when the situation is categorically different.

I agree it's getting "tiring" but this is another common human trope. You see the same thing too many times and you lose the sense of danger that comes with it. It's like sky diving all the time. You eventually lose the fear via repetition, but fundamentally speaking, you are still parachuting out of a fucking plane.

Fwiw, the AI companies have been saying a lot about these questions should be answered. Whether you want their answers is another story.

You're unhappy about the labs; I think you're just not ready to accept their rule; you consider them nothing but scrappy startups, which they are. But power is power, like it or not.

Am I personally happy with any of this? Does it matter?

ChatGPT and Claude are literally building staging scripts for me as I type this.

5 years ago, few people on this planet would have thought the computer would make advanced POSIX compliant shell scripts, but here we are.

The same is even more true of our intelligence. We're building computers with the size and power consumption characteristics of entire cities to do things that may almost, but not quite, match what our brains do with a kilogram and a half of mass and about 20 watts at the top end of power consumption.

The only way we are ever going to match that with technology is to run AI workloads on human brain tissue, which Rick-and-Morty level horror is being actively worked on as I understand it. The original concept for The Matrix wherein the machines used humans to run compute workloads on their brains actually kind of makes sense.