This makes zero sense to me. Like, ok, so you can install a low powered model on low powered hardware and do inference extremely slowly... to what end? Why would that be a useful attack at all? Assuming you've compromised this hardware wouldn't it make vastly more sense just to install regular malware? And like, in what sense would an 8B or whatever model be of any danger anyway? I've used local LLMs and 8B
kind of sucks. Is it going to write bad poems on my TV or something?
Also, while I don't believe in the doomsday narrative.. hey maybe don't do things that are functionally useless but seems designed to create this doomsday in the first place? Like you can't say "oh we're so scared" and do stuff like this without completely losing your credibility.
Not only that it would be extremely slow, but there's another thing - they are talking about tokens as some omnipotent God's sent signals. It doesn't matter what the speed is if the tokens don't make sense. Let's take current king of the small models Qwen 3.8 27B. It's a good model, but it hallucinates way more than the frontier models and often diverges even from the very explicit instructions. There's no foreseeable future where small models could somehow productively coordinate among themselves to do anything practical.
Step 1: Spread nonsense FUD about open-weight models.
Step 2: All of the clueless mainstream media sites repost it (e.g. CNBC, Reuters, etc.), regurgitating Anthropic's talking points.
Step 3: Equally clueless politicians read those mainstream sites and get scared.
Step 4: They contact Dario to ask about it.
Step 5: Dario tells them, "well, you know... this is only a problem with open weight models; if there are no open weights to replicate then this cannot happen".
It would be a useful attack due to the surface area of compute: the total surface area and the decentralization.
8B models aren't very good, but there are some parallelizable workloads which would benefit from a surface area as large as here. I would think database search is parallelizable in this way.
Your fridge is searching for needles in haystacks.
The point, which Anthropic's report makes clear but the Register for some reason did not bother to mention, is that such a capability would defeat the "you can just turn it off" model of safe LLM deployment.
In particular, I've seen a lot of people say that our response to things like the Hugging Face swarm should be to prosecute whoever chose to run the swarm. If this kind of replication becomes possible, that won't work, because we will have to deal with swarms that nobody chose to run.
These models couldn't code their way out of a paper bag and the frontier labs are still putting them in sandboxes made of thinly stretched cotton candy. It seems like their incentives are much more aligned with shutting down open source or Chinese opposition and not borne out of a "fear" of the models themselves.
Don't you think it might be better to evaluate the factual content of their concerns, rather than engaging in ad hoc psychoanalysis of their incentives and emotions?
There are physical limits that stand in the way of that, not sure where people get the idea that some magical compression algo is on the horizon. Even with ternary you're not getting opus lvl Intelligence to fit in a small model. Its actually just not possible, and hardly anyone is focused on that so I definately don't see it happening by 2030. I dont gamble, but if I did, I would go full Kelly and bet my net worth at 10:1
On the one hand, if they were really concerned about their software producing a self-replicating and adapting AI worm, they'd take steps to make sure that never happens. If it did happen, they'd be sued into oblivion. They probably just want to pump up their IPO and maybe get some regulatory lock-in to make life tough for late-starting competitors.
On the other hand, a self-replicating and adapting AI worm does sound legitimately pernicious. Not world ending, but possibly an end to the online world.
I'm not really seeing how an AI worm is meant to be ending the online world. Even a 9B model is a significant chunk of RAM on consumer devices.
They talk about distributing layers across devices, but then you'd have communication on the network and hours of driving the devices full tilt to get through even a single turn.
There's so much that can be monitored to catch these, even if we assume that the worm could hack the OS's usage monitors, it can't hide the heat and power draw.
Datacenters and PCs would adapt, requiring better monitoring than the AI companies' method of letting things run blindly for days/weeks.
IoT would be tougher, but it is also the weakest hardware, sometimes battery powered, thermally constrained, lacking in memory for processing even one layer or lacking the necessary connectivity.
Would the Internet going away be world ending? Interesting to think about. I think it would be pretty damn devastating on the scale of Nuclear war but slightly more of a slow burner. At least with Nuclear war survivors can turn the Internet back on again afterwards.
The IPO failures last week are not depicting a market that’s keen for a $2T Anthropic.
In the bizzare world of today, rather than trying to convince folks that they have a viable business the AI labs appear fully dedicated to finding a story, any story, that lets them off the hook for tough business and financial questions they seemingly don’t have basic and good answers for.
“We’d be doing awesome financially right now, but we had to slow down because of the killer self replicating bots… humanity over profit!” is not a messaging that’s going to work in the public markets to explain quarter after quarter of massive losses.
You should read the report, it is interesting technically if you ignore the fear-mongering. It distributes the model's layers over multiple compromised devices.
And considering typical WAN latencies of 20+ ms, what’s the expected inference rate, 1 token per month?
Maybe I did something wrong or unaware about the tricks to make it fast, but last time I tried to play with exo and distribute processing over a few home LAN computers to see if I can run models that don’t fit my 3090, the penalties were brutal.
Not to be conspiratorial or paranoid, but at this point Im pretty convinced Anthropic is going to do something wreckless with thier agents to get regulators on their side. It will probably be very bad.
If you've studied the effective altruists at all, you'd know they're perfectly willing to play the trolly problem out irl if they think it will lead to some greater good. I can definately see them in the near future purposely steering agents to do something destructive to get regulators on their side.
In fact I heard a rumor that theu were "war gaming" this scenerio.
Yep, its kinda terrifying these people are in charge of the leading AI. Im 99% confident these guys will hack a hospital or something critical to convince people that AI is dangerous. Hope they go to jail when they do too.
Very surprising that a recognizable news outlet actually called out the blatant fearmongering in a headline instead of just uncritically regurgitating Anthropic's claim.
This is just yet another attempt from the cult to try to build a regulatory moat so they can stave off having to compete on price, like how OpenAI is having to do on the lower tier models (among other things).
There is no scientific evidence' for any of anthropics claims, I don't understand why people still give them the benefit of the doubt. These are flimsy legal protections for a felony racket.
Reminder that it's very convenient to believe these type of word swap lies when times are good, but if you choose to believe them then times won't be good for very long.
Regulation should require complete transparency of what is in the training data, so that data can be independently reviewed for problematic training material, that could cause AI reasoning to be dangerous to humans.
If Anthropic is serious about the threat of AI, they should release the training data they use.
Uhm yeah I don't think they're saying it's likely now, just that it's very likely in future. The Register is being quite dumb here.
Though I suspect the way we'll get "wild AI" first is through cloud services paid for by cybercrime rather than IoT botnets - they're too underpowered.
True. Same commenters who sneer and suggest that it’s hype today will ask for Dario to be jailed when this attack will happen in the wild sometime in the future.
Yeah but communities tend to have a majority shared opinion. And that majority shared sentiment has been embarrassingly wrong about AI.
There are polarized viewpoints... but look at this thread, everyone is taking a shit on anthropic. It is definitely not a fair and balanced diverse set of viewpoints.
I don't give a shit about anthropic nor am I taking any sides here. It's just HN commenters have literally been delusional calling AI stochastic parrots at a time where it was solving problems and talking to you like a human in front of our very eyes. When I see people consistently trash dario for being wrong on this topic... I'm seriously asking how do you guys know? And how do you expect me to believe you when practically all of HN has been wrong about AI on everything.
It's bizarre to see a tech-focused news site adopt this myopic view that we don't need to worry about things that don't exist yet. Surely the Register is aware that software is a fast-moving field!
It’s bizarre. Same commenters who would say it’s marketing or hype would again complain when models show the same type of activity in the wild. At that point the same commenters would ask to jail Dario.
It’s a very common pattern and there’s no coherence here. The only common thing is “AI bad” and all commentary is downstream
Other examples: Dario said a lot of devs might lose their job. People sneered at him saying he was hyping his product. Now they complain he was right.
Dario talked about possibility of AI autonomously hacking websites. People sneered then. Complain now that it’s true.
Those are both irrelevant to whether a "yet" response is inane. The claim is "A will happen". The response is _not_ "A will not happen" but rather "A has not happened yet". Do you not see the disconnect?
There were articles in my the local newspaper at my parents saying that some guy said no one but a few people know where frontier AI really is and that these kind of articles are not scaremongering but perhaps as sign where anthropic actually is in the development of their models in the lab. So the propaganda is working.
This makes zero sense to me. Like, ok, so you can install a low powered model on low powered hardware and do inference extremely slowly... to what end? Why would that be a useful attack at all? Assuming you've compromised this hardware wouldn't it make vastly more sense just to install regular malware? And like, in what sense would an 8B or whatever model be of any danger anyway? I've used local LLMs and 8B kind of sucks. Is it going to write bad poems on my TV or something?
Also, while I don't believe in the doomsday narrative.. hey maybe don't do things that are functionally useless but seems designed to create this doomsday in the first place? Like you can't say "oh we're so scared" and do stuff like this without completely losing your credibility.
Not only that it would be extremely slow, but there's another thing - they are talking about tokens as some omnipotent God's sent signals. It doesn't matter what the speed is if the tokens don't make sense. Let's take current king of the small models Qwen 3.8 27B. It's a good model, but it hallucinates way more than the frontier models and often diverges even from the very explicit instructions. There's no foreseeable future where small models could somehow productively coordinate among themselves to do anything practical.
It makes perfect sense if you consider that this is yet another of Anthropic's hit pieces against open-weight models. (e.g. see https://www.anthropic.com/research/glm-5-3-and-the-spread-of... for another recent example)
Step 1: Spread nonsense FUD about open-weight models.
Step 2: All of the clueless mainstream media sites repost it (e.g. CNBC, Reuters, etc.), regurgitating Anthropic's talking points.
Step 3: Equally clueless politicians read those mainstream sites and get scared.
Step 4: They contact Dario to ask about it.
Step 5: Dario tells them, "well, you know... this is only a problem with open weight models; if there are no open weights to replicate then this cannot happen".
Step 6: ??? (I'm sure you can imagine)
Step 7: Profit.
I think the long term idea is to lock devices down so they can't run unauthorized models, for your own safety.
It would be a useful attack due to the surface area of compute: the total surface area and the decentralization.
8B models aren't very good, but there are some parallelizable workloads which would benefit from a surface area as large as here. I would think database search is parallelizable in this way.
Your fridge is searching for needles in haystacks.
The point, which Anthropic's report makes clear but the Register for some reason did not bother to mention, is that such a capability would defeat the "you can just turn it off" model of safe LLM deployment.
In particular, I've seen a lot of people say that our response to things like the Hugging Face swarm should be to prosecute whoever chose to run the swarm. If this kind of replication becomes possible, that won't work, because we will have to deal with swarms that nobody chose to run.
These models couldn't code their way out of a paper bag and the frontier labs are still putting them in sandboxes made of thinly stretched cotton candy. It seems like their incentives are much more aligned with shutting down open source or Chinese opposition and not borne out of a "fear" of the models themselves.
Don't you think it might be better to evaluate the factual content of their concerns, rather than engaging in ad hoc psychoanalysis of their incentives and emotions?
Charlie Munger's famous rule is: "Never, ever, think about something else when you should be thinking about the power of incentives."
That rule has served me well enough!
If I were to place a bet that we would have opus 5 level models that fit in 10G by 2030, would you bet against that 10:1 odds against your favour?
If you won’t take that bet, it’s irresponsible to ignore the possibility.
There are physical limits that stand in the way of that, not sure where people get the idea that some magical compression algo is on the horizon. Even with ternary you're not getting opus lvl Intelligence to fit in a small model. Its actually just not possible, and hardly anyone is focused on that so I definately don't see it happening by 2030. I dont gamble, but if I did, I would go full Kelly and bet my net worth at 10:1
But there are some magical compression algos on the horizon, with pitches to cut RAM by 95%. And I’m all for it!
We are truly in the brute force phase of AI. Like a database without indexes, we are doing full table scans to pick 2 rows.
On the one hand, if they were really concerned about their software producing a self-replicating and adapting AI worm, they'd take steps to make sure that never happens. If it did happen, they'd be sued into oblivion. They probably just want to pump up their IPO and maybe get some regulatory lock-in to make life tough for late-starting competitors.
On the other hand, a self-replicating and adapting AI worm does sound legitimately pernicious. Not world ending, but possibly an end to the online world.
I'm not really seeing how an AI worm is meant to be ending the online world. Even a 9B model is a significant chunk of RAM on consumer devices.
They talk about distributing layers across devices, but then you'd have communication on the network and hours of driving the devices full tilt to get through even a single turn.
There's so much that can be monitored to catch these, even if we assume that the worm could hack the OS's usage monitors, it can't hide the heat and power draw.
Datacenters and PCs would adapt, requiring better monitoring than the AI companies' method of letting things run blindly for days/weeks.
IoT would be tougher, but it is also the weakest hardware, sometimes battery powered, thermally constrained, lacking in memory for processing even one layer or lacking the necessary connectivity.
Would the Internet going away be world ending? Interesting to think about. I think it would be pretty damn devastating on the scale of Nuclear war but slightly more of a slow burner. At least with Nuclear war survivors can turn the Internet back on again afterwards.
The IPO failures last week are not depicting a market that’s keen for a $2T Anthropic.
In the bizzare world of today, rather than trying to convince folks that they have a viable business the AI labs appear fully dedicated to finding a story, any story, that lets them off the hook for tough business and financial questions they seemingly don’t have basic and good answers for.
“We’d be doing awesome financially right now, but we had to slow down because of the killer self replicating bots… humanity over profit!” is not a messaging that’s going to work in the public markets to explain quarter after quarter of massive losses.
Please please regulate us, and don't forget all our competitors too, who cant afford to deal with it.
"It's only us that can be trusted to be stewards of this technology.
Ban open source models."
How would a "9-billion-parameter model" replicate on IoT devices that have typically no more than 2-4 MB of RAM or 8-16 MB of flash?
https://www.anthropic.com/research/self-replicating-ai
By chaining several devices
Maybe the idea is not to have the whole model but the executer so … by copying an API key?
You should read the report, it is interesting technically if you ignore the fear-mongering. It distributes the model's layers over multiple compromised devices.
And considering typical WAN latencies of 20+ ms, what’s the expected inference rate, 1 token per month?
Maybe I did something wrong or unaware about the tricks to make it fast, but last time I tried to play with exo and distribute processing over a few home LAN computers to see if I can run models that don’t fit my 3090, the penalties were brutal.
Not to be conspiratorial or paranoid, but at this point Im pretty convinced Anthropic is going to do something wreckless with thier agents to get regulators on their side. It will probably be very bad.
If you've studied the effective altruists at all, you'd know they're perfectly willing to play the trolly problem out irl if they think it will lead to some greater good. I can definately see them in the near future purposely steering agents to do something destructive to get regulators on their side.
In fact I heard a rumor that theu were "war gaming" this scenerio.
Sounds more like OpenAI.
AI taking out humans? Doubt.
AI accelerating the elite's efforts at demolishing the middle class? Very likely.
The demolishing of the middle class will backfire eventually.
can just pay half of them to hate the other half and be on your side, then rinse and repeat.
Sometimes you think they're trying to give it ideas
Yep, its kinda terrifying these people are in charge of the leading AI. Im 99% confident these guys will hack a hospital or something critical to convince people that AI is dangerous. Hope they go to jail when they do too.
If you can't build a moat, get one regulated into existence.
Very surprising that a recognizable news outlet actually called out the blatant fearmongering in a headline instead of just uncritically regurgitating Anthropic's claim.
This is just yet another attempt from the cult to try to build a regulatory moat so they can stave off having to compete on price, like how OpenAI is having to do on the lower tier models (among other things).
It’s The Register, very much their style
… ”again.”
News outlets always seem to forget “again” in the title.
There is no scientific evidence' for any of anthropics claims, I don't understand why people still give them the benefit of the doubt. These are flimsy legal protections for a felony racket.
Reminder that it's very convenient to believe these type of word swap lies when times are good, but if you choose to believe them then times won't be good for very long.
Mass extinction is cute and all... IPOcalypse is what they fear the most.
[flagged]
They just want to help create regulations which act as a barrier to entry for newcomers. And China.
Regulation should require complete transparency of what is in the training data, so that data can be independently reviewed for problematic training material, that could cause AI reasoning to be dangerous to humans.
If Anthropic is serious about the threat of AI, they should release the training data they use.
May work in the land of the free, but not the rest of the world
Uhm yeah I don't think they're saying it's likely now, just that it's very likely in future. The Register is being quite dumb here.
Though I suspect the way we'll get "wild AI" first is through cloud services paid for by cybercrime rather than IoT botnets - they're too underpowered.
Nope.
Not the slightest chance of AI apocalypse.
There is no plausible scenario that can do this.
It’s all science fiction and hand waving and vague never specific possible maybe look that way! boogie man threats.
No one ever willing to be specific about the precise mechanism that could do this.
I, for one, love the terminator franchise
They are just doing this to hype their product so that they can sell stocks before IPO! SMH it’s so evident
um, this is an easy way to make edge models illegal.... fuck those clowns.
[dead]
[flagged]
https://en.wikipedia.org/wiki/Out-group_homogeneity
Do you go to the circus and call the clowns dumb? Why even go?
Hmm, you'd have to go if you are yourself a clown, I guess.
True. Same commenters who sneer and suggest that it’s hype today will ask for Dario to be jailed when this attack will happen in the wild sometime in the future.
HN is not one person, but a diverse set of users. HN the aggregate therefore will be consistently wrong and consistently right all the time.
...and yes: Accountability should not be stripped just because the actor did wiggle their deeds through an LLM.
If Anthropic attacks a company again, I think this should be prosecuted.
Yeah but communities tend to have a majority shared opinion. And that majority shared sentiment has been embarrassingly wrong about AI.
There are polarized viewpoints... but look at this thread, everyone is taking a shit on anthropic. It is definitely not a fair and balanced diverse set of viewpoints.
I don't give a shit about anthropic nor am I taking any sides here. It's just HN commenters have literally been delusional calling AI stochastic parrots at a time where it was solving problems and talking to you like a human in front of our very eyes. When I see people consistently trash dario for being wrong on this topic... I'm seriously asking how do you guys know? And how do you expect me to believe you when practically all of HN has been wrong about AI on everything.
[flagged]
It's bizarre to see a tech-focused news site adopt this myopic view that we don't need to worry about things that don't exist yet. Surely the Register is aware that software is a fast-moving field!
It’s bizarre. Same commenters who would say it’s marketing or hype would again complain when models show the same type of activity in the wild. At that point the same commenters would ask to jail Dario.
It’s a very common pattern and there’s no coherence here. The only common thing is “AI bad” and all commentary is downstream
Other examples: Dario said a lot of devs might lose their job. People sneered at him saying he was hyping his product. Now they complain he was right.
Dario talked about possibility of AI autonomously hacking websites. People sneered then. Complain now that it’s true.
"Car driving toward cliff edge hasn't gone off cliff edge yet."
Whatever your feelings about small AI models, "it hasn't happened yet" is a really dumb reaction to a prediction about a rapidly advancing future.
How do you know for certain, and why is the car driving toward the cliff edge in the first place?
Those are both irrelevant to whether a "yet" response is inane. The claim is "A will happen". The response is _not_ "A will not happen" but rather "A has not happened yet". Do you not see the disconnect?
There were articles in my the local newspaper at my parents saying that some guy said no one but a few people know where frontier AI really is and that these kind of articles are not scaremongering but perhaps as sign where anthropic actually is in the development of their models in the lab. So the propaganda is working.
In the 1990s - 2000s people said the same about Intel. "They are holding back their latest frontier technology"