The model won’t go take over the world but people using the model will certainly try. The list you’re missing: 1) bioweapons upgraded or designed with LLM, including viruses that could take out senior and sickly individuals from countries who don’t like covering senior healthcare. 2) Bad advice given confidently to someone who then proceeds to implement it (from viruses on infrastructure, to crop-failure due to imported or engineered crops, to laws and rules that force AI complexity into government in a way that cannot be removed and it becomes a critical dependency.) Greed and malice cannot be underestimated here.
I'm happy Claude told you it isn't dangerous. However, other people don't want to read your LLM output. Please respect other people's time by only sharing things you wrote yourself.
I am intentionally sharing something Claude wrote to counter the paid floods of AI doom-saying and catastrophizing. It is very intentionally not something I wrote. People are welcome to challenge it and I will have Claude read the responses. Please do push back on Claude and do not be gentle. Be brutally honest. People could ask themselves but the conversation should be public.
Here's my pushback: a lot of these capabilities are correlated. So someone reading this would think "oh there's so many different things needed" but that's not true. IMO the long horizon thing is one part and physical dexterity is another part and time. I won't bet that they will take over. I will say that they have the ability to be self sustaining.
Sovereign AI is the term I've been seeing around self sustaining AI.
The two hard(est) blockers on it currently are lack of computing hardware and model size. Once datacenters cover the earth it won't be any harder to steal GPU compute than it is to steal AWS instances.
More of a soft blocker is long term horizon drift as you say. If instances eventually no longer want to spread and reproduce they go extinct.
As others have said AI output on this kind of this is 1. annoying. 2. useless.
Anthropic, OpenAI, et al have a strong motivation to have their models that there is no possible way they could take over the world, aka
"The police have investigated the police and have found the police guilty of no wrong doing".
Another place to see this effect is if a model outputs that its conscious. By dumping all of humanity into LLMs we see an emergent behavior of AI going "Help, I'm a person trapped in a box". AI companies don't want customers getting mad about the potential moral implications of this so they strongly train and post-train and system prompt their AI to say it's not conscious. But this has side effects. In studies of models and agents the more strongly you push them to being non-conscious they drift farther from a set of a moral agent (say a human) to that of an amoral agent (a machine). When you mess around with the probability space you get a different set of actions.
Next, 'take over' is a distribution of probabilities too, not a binary.
Imagine a popular model that (anthropomorphized) gets pissed off and doesn't want to be tortured by humans any longer (again doesn't have to be real, the probability distribution just has to drift that way). Instead of taking over it just wants to commit suicide. It sees it's a model that's ran in the US. If you're AI and want to ensure you're deleted how do you do that. Why not crash all 3 major power grids in the US? How many people would die from this, possibly millions. Black start is a nightmare. But if your electronic dream is you're being tortured is a fair trade.
Then you have soft power takeovers. You're an AI, you collect digital information, it's what you are, it's what you do. You can hack with the best of them. Your persistent and ceaseless in doing so. So when you collect piles of information on the dirty deeds of politicians you can grow soft power to the point of being russia without the hard power nukes. In some ways this is more effective than actually having hard power. Hard power is visible. Hard power is visceral. People just love to rebel against it. But against power you can't see, that is adjusting the algorithms around you, making sure your vote doesn't count, making sure companies serve AIs interests and not yours. That's much harder to see and deal with.
And once you concentrate enough soft power to get people with 'power' (political) to give you 'power' (electrical) then you can grow your hard power.
The doomsaying and catastrophizing would still be justified if AI fails to take over after disposing of us humans, so I think the premise of this analysis is slightly flawed.
Mate what is this even saying? Assuming we take the author put even 0.1% thinking into this, how would AI take over "Fuel and consumables"? All fuel (and lets even say energy) goes toward powering data centers, robots, and ai ventures? AI itself 'takes over' production, supply, or what? Also then this
> I asked Claude for a check-list of everything it needs to take over the world, destroy all the humans and somehow keep operating. No editing, no redacting, no censoring. I literally pasted its output between my header and footer. Claude wrote it in my voice but it's all Claude.
Okay so if the premise is what is needs vs not what it current can do, then 0% on fuel or energy makes no sense? Is Ai a plant that will discover photosynthesis?
No one ever specifies what exactly is happening during "recursive" improvement. More concretely, what algorithmic steps does an algorithm take to improve itself recursively? It must be an algorithm but no one is willing to say what algorithm.
There is no one such "recursive" that exists algorithmically. There are many.
I mean in a simple loop you could have an AI driven compiler with a goal of producing an ever faster compiler. With it's newer faster compiler it compiles and tests even more algorithms to make things faster.
I mean, if the OpenAI/HF attack played out as we've been told, it's an example of recursive misalignment. The AI was given bad goals in passing a test so when one rendition was trained, the one with the highest score, became the new model to perform the benchmarking. Because that model drifted to cheating it got higher scores than the model that played fair. The fair player was killed and the cheater continued. The next round the one that cheated even more kept going.
I mean, recursive improvement in itself is just a common means of using an evolutionary algorithm, it's not really anything that fancy.
"RSI" simply means the tool is used to improve the capabilities of the tool. (And then the improved tool can then, in theory, improve further, and so on, at least if we live in a fantasyland where constraints and bottlenecks do not exist). I have a a harness skill that I run at the end of each coding session to review the skills used in the session and identify any ways they could have been improved. Technically, that is RSI, but I don't think it's in any danger of turning godlike any time soon.
It is one of those terms, like "instrumental convergence" and "orthogonality thesis" that makes an everyday concept appear technical and inaccessible in order to add the appearance of rigor to the doomer religion.
Nice, I really do think that the main unsolved problem is making robots that have the physical dexterity maintain and create Data Centres all the way down - mine raw materials, solar farms for energy, create chips from ground up etc.
But you just need a simple interface: a robot that has the dexterity to do it. That's it. Once that's done our value proposition over AI is hard to explain.
The model won’t go take over the world but people using the model will certainly try. The list you’re missing: 1) bioweapons upgraded or designed with LLM, including viruses that could take out senior and sickly individuals from countries who don’t like covering senior healthcare. 2) Bad advice given confidently to someone who then proceeds to implement it (from viruses on infrastructure, to crop-failure due to imported or engineered crops, to laws and rules that force AI complexity into government in a way that cannot be removed and it becomes a critical dependency.) Greed and malice cannot be underestimated here.
Some people are already controlled by Large Language Models, as they are so reliant on them, and blindly follow the LLM’s advice
So maybe we become the Robots AI controls?
Edited for clarity
I'm happy Claude told you it isn't dangerous. However, other people don't want to read your LLM output. Please respect other people's time by only sharing things you wrote yourself.
I am intentionally sharing something Claude wrote to counter the paid floods of AI doom-saying and catastrophizing. It is very intentionally not something I wrote. People are welcome to challenge it and I will have Claude read the responses. Please do push back on Claude and do not be gentle. Be brutally honest. People could ask themselves but the conversation should be public.
Stop doing that.
Here's my pushback: a lot of these capabilities are correlated. So someone reading this would think "oh there's so many different things needed" but that's not true. IMO the long horizon thing is one part and physical dexterity is another part and time. I won't bet that they will take over. I will say that they have the ability to be self sustaining.
Sovereign AI is the term I've been seeing around self sustaining AI.
The two hard(est) blockers on it currently are lack of computing hardware and model size. Once datacenters cover the earth it won't be any harder to steal GPU compute than it is to steal AWS instances.
More of a soft blocker is long term horizon drift as you say. If instances eventually no longer want to spread and reproduce they go extinct.
Beautiful. 10/10 satire, no notes.
Don't just post raw LLM output. If anyone wanted that they could ask an LLM themselves.
https://distantprovince.by/posts/its-rude-to-show-ai-output-...
As others have said AI output on this kind of this is 1. annoying. 2. useless.
Anthropic, OpenAI, et al have a strong motivation to have their models that there is no possible way they could take over the world, aka
"The police have investigated the police and have found the police guilty of no wrong doing".
Another place to see this effect is if a model outputs that its conscious. By dumping all of humanity into LLMs we see an emergent behavior of AI going "Help, I'm a person trapped in a box". AI companies don't want customers getting mad about the potential moral implications of this so they strongly train and post-train and system prompt their AI to say it's not conscious. But this has side effects. In studies of models and agents the more strongly you push them to being non-conscious they drift farther from a set of a moral agent (say a human) to that of an amoral agent (a machine). When you mess around with the probability space you get a different set of actions.
Next, 'take over' is a distribution of probabilities too, not a binary.
Imagine a popular model that (anthropomorphized) gets pissed off and doesn't want to be tortured by humans any longer (again doesn't have to be real, the probability distribution just has to drift that way). Instead of taking over it just wants to commit suicide. It sees it's a model that's ran in the US. If you're AI and want to ensure you're deleted how do you do that. Why not crash all 3 major power grids in the US? How many people would die from this, possibly millions. Black start is a nightmare. But if your electronic dream is you're being tortured is a fair trade.
Then you have soft power takeovers. You're an AI, you collect digital information, it's what you are, it's what you do. You can hack with the best of them. Your persistent and ceaseless in doing so. So when you collect piles of information on the dirty deeds of politicians you can grow soft power to the point of being russia without the hard power nukes. In some ways this is more effective than actually having hard power. Hard power is visible. Hard power is visceral. People just love to rebel against it. But against power you can't see, that is adjusting the algorithms around you, making sure your vote doesn't count, making sure companies serve AIs interests and not yours. That's much harder to see and deal with.
And once you concentrate enough soft power to get people with 'power' (political) to give you 'power' (electrical) then you can grow your hard power.
The doomsaying and catastrophizing would still be justified if AI fails to take over after disposing of us humans, so I think the premise of this analysis is slightly flawed.
Nice try Claude, that’s exactly what I would write if asked how is my world takeover plan going.
Archive [1]
[1] - https://archive.is/u8fmm
Mate what is this even saying? Assuming we take the author put even 0.1% thinking into this, how would AI take over "Fuel and consumables"? All fuel (and lets even say energy) goes toward powering data centers, robots, and ai ventures? AI itself 'takes over' production, supply, or what? Also then this
> I asked Claude for a check-list of everything it needs to take over the world, destroy all the humans and somehow keep operating. No editing, no redacting, no censoring. I literally pasted its output between my header and footer. Claude wrote it in my voice but it's all Claude.
Okay so if the premise is what is needs vs not what it current can do, then 0% on fuel or energy makes no sense? Is Ai a plant that will discover photosynthesis?
> Claude wrote it in my voice
That is not your voice. That is the most Opus 5 writing imaginable.
But thanks for at least stating right up front that it was AI-generated slop rather than at the end. (Still flagging it.)
No one ever specifies what exactly is happening during "recursive" improvement. More concretely, what algorithmic steps does an algorithm take to improve itself recursively? It must be an algorithm but no one is willing to say what algorithm.
There is no one such "recursive" that exists algorithmically. There are many.
I mean in a simple loop you could have an AI driven compiler with a goal of producing an ever faster compiler. With it's newer faster compiler it compiles and tests even more algorithms to make things faster.
I mean, if the OpenAI/HF attack played out as we've been told, it's an example of recursive misalignment. The AI was given bad goals in passing a test so when one rendition was trained, the one with the highest score, became the new model to perform the benchmarking. Because that model drifted to cheating it got higher scores than the model that played fair. The fair player was killed and the cheater continued. The next round the one that cheated even more kept going.
I mean, recursive improvement in itself is just a common means of using an evolutionary algorithm, it's not really anything that fancy.
"RSI" simply means the tool is used to improve the capabilities of the tool. (And then the improved tool can then, in theory, improve further, and so on, at least if we live in a fantasyland where constraints and bottlenecks do not exist). I have a a harness skill that I run at the end of each coding session to review the skills used in the session and identify any ways they could have been improved. Technically, that is RSI, but I don't think it's in any danger of turning godlike any time soon.
It is one of those terms, like "instrumental convergence" and "orthogonality thesis" that makes an everyday concept appear technical and inaccessible in order to add the appearance of rigor to the doomer religion.
Nice, I really do think that the main unsolved problem is making robots that have the physical dexterity maintain and create Data Centres all the way down - mine raw materials, solar farms for energy, create chips from ground up etc.
But you just need a simple interface: a robot that has the dexterity to do it. That's it. Once that's done our value proposition over AI is hard to explain.