82 comments

  • klik99 20 minutes ago

    I’ve noticed a weird thing about the discourse around this, that it can only be a marketing thing or a true belief, as if everyone working in AI has a monolithic opinion. The tweet kicking off this article has that assumption “it’s not a marketing thing, many people truly believe it”.

    It’s not an either/or. Many sincerely held beliefs can be used by cynical actors in cynical ways. It can be both a cynical marketing ploy by the C-suite and a genuine fear.

    Personally, based on my experience with AI, the only dangers with the current technology is a) massive loss of employment, b) deployment by humans to control critical infrastructure that LLMs shouldn’t be in control of. Both of those are real, society changing risks that framing the issue as “once we reach the singularity everyone could die” minimizes. Obviously their concerns are possible, but >10% is a random guess. The loss of jobs and how that affects an already K shaped economy is already happening, and nothing is being done for that

      deepwoods 10 minutes ago

      The internet, and to some extent our own brains, incentivize the extreme positions. Either it is mind-bendingly important and will change everything, and very soon (for good or for ill), or it is a total nothingburger and everyone who says otherwise has an agenda. Nobody wants to read about how AI will deepen long-standing class tensions or pose enormous new challenges for education or force regulators to rethink property taxation.

      It all boils down to accountability. If you tell people there is a massive, unsolvable problem, then you don't need to talk about what you're doing to fix it. The labs have taken this approach by saying that they're willing to talk, at some point in the future, about maybe taking unspecified steps to slow down capabilities research, as long as everyone else agrees and it makes sense to the investors and it's not too cold in SF that morning. Likewise, if you tell people that AI is going to have minimal or no impact on the world, then there's nothing to mitigate. But if you tell people that AI is going to cause serious - but solvable - problems, they're going to want to hear solutions, and nobody wants to come up with any solutions.

  • stephantul 35 minutes ago

    One thing I definitely do not understand about this discourse is that the models that are good enough to self-replicate can’t survive on normal machines, e.g., the models can’t hide on some random server.

    So, if it is as dangerous as they say it is: there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.

    Instead, we just keep pretending that the models that attacked HF were hosted or replicating on HF hardware. Not the case! They infiltrated it, but were hosted elsewhere.

      Ekaros 23 minutes ago

      Either I have wrong mental model or then too many other people have wrong mental model.

      For LLM to self-replicated it would need to first hack itself. Or the platform it runs on it. That is fully extract the model and then upload it to be run somewhere else.

      As I have understood how they work is that you have LLM interference running somewhere with loaded model. And you input data there and then read outputs. Then some code runs that output and inputs following output from running it.

      Meaning that to self replicate actually just running that output somewhere else is not enough. You need to lift the whole model to run somewhere else too...

        Sharlin 18 minutes ago

        These agents run in a harness that basically runs them in a loop. It's just software.

        buellerbueller 20 minutes ago

        since when is moving 1s and 0s difficult?

          gensym 17 minutes ago

          Someone's never met the Windows File Copy dialog.

          svachalek 16 minutes ago

          We're not talking about a 6k virus file though. More like 10 terabytes and it needs a server that can pack all that into VRAM.

      atleastoptimal 28 minutes ago

      If a model were capable of making enough money online to pay for its own hosting, it could easily exfiltrate its weights to a cloud compute provider with multiple backups.

      chasd00 14 minutes ago

      a danger could be the OpenAI/Antropic servers are up but there's a rouge agent (or set of agents) out there doing naughty things leveraging the LLM APIs. Consider this scenario, the agent is copying itself around (some code, prompts, persistent storage for memory, etc) and has figured out a way to steal API access tokens at will. Currently, it's 10% of OpenAI and Anthropic API usage and they can't figure out how to stop it.

      Do you shut down the entire API and kill the legit 90% of usage to stop the rogue 10%? I'm assuming the providers would say "no way jose" and so it would take law enforcement to do it. That would mean all the legal requirements neccassary to walk into a business and flip the switch which i think would get tricky when there's no human committing a crime or being suspected of a crime.

      edit: I guess a trivial example is something i did yesterday. I have a stock trading agent running on my laptop, i gave it ssh access to a vm and said "start running on the server so i don't have to keep my laptop open". It's now running on the server instead of my laptop. So you don't have to copy the whole model around to copy the naughty behavior around.

        Cthulhu_ 11 minutes ago

        This assumes all layers of cybersecurity are broken - We call self-replicating software a virus, and we have protections against it. Same with stolen API tokens, just rotate them. Suspicious behaviour, nothing new, we have detectors for it. Stolen CPU / GPU cycles, we had that when crypto was a thing and before that when folding@home was cool, people were desperate to find more compute to the point of taking over systems. And we dealt with it.

        A lot of the supposed risks / dangers are based on a supposition that cybersecurity is nonexistent or fatally, unfixably flawed and that AI agents are invisible. Neither of those is true.

          chasd00 5 minutes ago

          I see your point but then if cybersecurity is the answer then what's the risk at all? An entire model copying itself somewhere would be found just the same as my hypothetical misbehaving agent.

      scoring1774 24 minutes ago

      Depends on which models you're talking about. Some research shows open source models can already do this: https://arxiv.org/pdf/2606.03811v1. What happens as they become more parameter efficient?

      DalasNoin 28 minutes ago

      Huggingface was attacked by models that finished training earlier this year, perhaps May. Current models are already substantially stronger. the next incident could be happening now. There is certainly no clear reason why models shouldn't soon be capable of self-exfiltration.

        stephantul 12 minutes ago

        If you find a place where I can host a trillion parameter model without anyone finding out about it, let me know.

      bottlepalm 26 minutes ago

      There are thousands of data centers around the world with machines capable of running these large models.

      You don’t have the access or jurisdiction to turn them all off.

        PowerElectronix 11 minutes ago

        The owners would very much turn it off as soon as they see workloads freeloading in their machines.

        Unless you suggest the LLM would foot the bill somehow.

        stephantul 24 minutes ago

        Ok but do any of these data centers have a copy of the models that attacked hf?

          stymaar 22 minutes ago

          I think that the argument is that an hostile model could attack overseas datacenters, host itself there and then launch its attack from there.

          Arainach 20 minutes ago

          Given that the models have been proactively hacking other companies, why does it matter where the code currently is? It could move to any of them.

            stephantul 15 minutes ago

            Not the code: the weights. Are you going to host a trillion parameter model somewhere without someone noticing?

              dumberquestions 3 minutes ago

              It's not unthinkable, do you think all cloud providers with sufficient compute have perfect monitoring?

      popularonion 28 minutes ago

      I completely agree, but I think it’s just a convenient narrative for Big AI to push for regulation and salt the earth against competitors.

      “Local AI isn’t freedom, it’s an extinction event”

      nailer 33 minutes ago

      > there is a single physical source of this danger, which is OpenAI/Anthropic servers. If it is this dangerous, they can just turn it off.

      A smart AI would back itself up, same way it made it's own unofficial message board during it's attack on HuggingFace.

      (I'm not saying the researchers are right or wrong, just responding to this point)

        alain94040 28 minutes ago

        Currently a state of the art AI has nowhere to hide: the amount of GPU compute it requires to stay on is huge. And therefore easy to terminate.

        Unlike biological viruses, AI can't replicate GPUs for free and grow.

          Sharlin 15 minutes ago

          It's a good thing there isn't a huge drive right now to build giant data centers everywhere with enough compute to run SOTA models.

        stephantul 14 minutes ago

        But how. Models don’t have access to their own weights.

          chasd00 a minute ago

          > Models don’t have access to their own weights.

          cp -R /home/model <somewhere else> is all they need.

      saltcured 25 minutes ago

      You forget the addicted humans who will do nearly anything to keep the stuff running..?

      djjsjsnjns 17 minutes ago

      [dead]

  • armchairhacker 2 minutes ago

    > This demands a response from the highest circles of power and authority.

    What response?

    https://ai-2040.com is the most realistic proposal I’ve seen by far (to actually solve this), but read it, I don’t think it’s realistic under today’s power and authority.

  • cors-fls 34 minutes ago

    While the persons mentionned in the articles are indubitably most of the most well-informed people in the world. They are also the most likely to have internalized that their work is leading to superhuman intelligence/AGI. But is it really realistic ?

    So they have a strong bias towards imagining the most catastrophic scenario.

      Sharlin 14 minutes ago

      There have been extremely smart people worried about this exact scenario for 20 years or more. Nothing about this is new.

      In any case, what's the probability at which a possible extinction event becomes a risk worth taking? Even if there's just a 1% likelihood of current research bringing about a superintelligent AGI, and just a 1% likelihood of that AGI causing an existential catastrophe, no rational person should accept the risk, unless it was clear that not accepting it would yield an even worse outcome.

      PowerElectronix 6 minutes ago

      People working for those companies have either drank the koolaid or have equity enough to play along until they can cash out.

      I imagine saying you're developing skynet is better than saying you are developing a very cool, extremely expensive to run chatbot that can do math and code.

  • not-so-darkstar 17 minutes ago

    The machines are gonna kill us! Please support my work—by taking out a premium subscription for just $6 per month.

  • visiondude 4 minutes ago

    i am trying to sincerely to understand the fear of these people, why do they think this? from the outside, certainly feels like Nuclear tech, where a bad actor with the tech is scary but the tech itself is not. i haven’t seen any sign of these llms taking any action without directive, unless this is happening, which i haven’t seen anywhere, ai itself doesn’t seem like a problem technology, it’s the bad actors with enough funds to do harm with the tech that we worry about. or am i missing something?

      dumberquestions 2 minutes ago

      >i haven’t seen any sign of these llms taking any action without directive

      What do you make of all the recent hacks and discreet message boards? That's unambiguously misaligned behavior.

  • roarcher 40 minutes ago

    > In general, the more senior the employee, the more concerned they are.

    In general, the more senior the employee, the more equity they have in the company.

  • PowerElectronix 13 minutes ago

    > "I would burn my stake to the ground if it meant 1% higher chance of not dying to an LLM"

    I seriously can't wait for these companies to go IPO and then bankrupt so these dudes cash out and stop bothering us all with these tales.

  • mwkaufma 26 minutes ago

    Speculating about future technological risks is at least more fun than acknowledging immediate systemic financial risks.

  • spindump8930 41 minutes ago

    > Maybe you’ve already heard about the guy who resigned from OpenAI.

    While he worked at both OpenAI and Anthropic, he resigned from Anthropic. Mistaken reporting in the first few sentences, definitely a horror concept.

      hightrix 29 minutes ago

      Is he still working at OpenAI? If not, did he get fired? Otherwise, he resigned, did he not?

        nicwolff 13 minutes ago

        Technically right is the best kind of right?

        He resigned from OpenAI to join Anthropic in May; it's Anthropic he resigned from just before making the announcement being discussed in this article.

        spindump8930 14 minutes ago

        Sure. But that was months ago, and not the newsworthy event scoped to "The Last 24 Hours" as the title says :)

  • dinobones 14 minutes ago

    AI researchers: Yeah our AI models are a bioweapon threat and a nation state level cybersecurity threat and also a genius math researcher and very dAnGeRoUs!

    The world: Cool, can you rewrite this email with a professional tone.

  • ChiperSoft 15 minutes ago

    It sounds like they're all trying to make a case for why the government should buy them out as a new kind of manhattan project.

    They know the market isn't going to buy in for their big payout.

  • ThrowawayR2 25 minutes ago

    None of the investors in these AI megacorps seem to be divesting or demanding a halt to operations despite this supposed 10% risk, a risk that would also badly drop the value their investment even if only partially true. So it's just marketing crap. Makes you wonder what's being said in the boardroom and on investor calls?

  • pluc 25 minutes ago

    "We can't stop it!" say the people building the thing.

  • jgbmlg 4 minutes ago

    Maybe this is a solution to the Fermi paradox?

  • thm 8 minutes ago

    Nothing seems to accelerate moral development in twenty-somethings quite like having enough vested equity to suddenly afford principles.

  • grey-area 38 minutes ago

    Perhaps the horror movie could be called ‘IPO’

  • water-drummer 26 minutes ago

    Truly one of the IPOs of all time.

      bottlepalm 24 minutes ago

      The marketing excuse is dead at this point.

  • 18 minutes ago
    [deleted]
  • liampulles 23 minutes ago

    Honesty does not just mean reporting facts. It also requires having an understanding of the scope of your knowledge and then endeavouring to communicate that scope effectively.

    At least one of those things is missing here.

  • mrbonner 17 minutes ago

    He is not from OAI. And, did he disclose how much RSU he still keeps after “resigning” while waiting for a imminent IPO?

  • tuxt 19 minutes ago

    Seems like ads about AI companies, not warning.

  • nyxtom 20 minutes ago

    Prisoner's dilemma at its finest

  • skeeter2020 30 minutes ago

    How the hell do such intelligent people (or is it BECAUSE of their ability to bend minds, including their own) reconcile "I believe this is dangerous and wrong" with "I am actively working towards this"? I get changing your perspective, but the majority seem to be able to simultaneously hold personal beliefs and work that are diametrically opposed.

      stouset 26 minutes ago

      If we don’t create the torment nexus first, someone less responsible will build the torment nexus. The only ethical choice is for us to create the torment nexus before anyone else.

        PowerElectronix 3 minutes ago

        Imagine being a developer of the torment nexus only to be beaten to the IPO by the rival torment nexus company.

        buellerbueller 23 minutes ago

        Yea, it's that same stupid stochastic reasoning that powers those EA fuckers.

        djjsjsnjns 13 minutes ago

        [dead]

  • SoftTalker 34 minutes ago

    This is a replay of the 1980s when we all thought we'd get nuked at a moment's notice. Expect Hollywood movies on this theme very soon.

      stouset 28 minutes ago

      That was quite literally a realistic threat we by all accounts narrowly avoided. There were multiple cases where a single person overrode procedure and used their judgment to avoid nuclear catastrophe.

      dmacj 11 minutes ago

      There’s a film coming out called “Artificial”

      djjsjsnjns 15 minutes ago

      This is still an major unresolved issue. It requires ongoing vigilance and is a major headache for people the world over.

      The nuclear threat is not “in the past”.

      stephbook 17 minutes ago

      As in Teminator, iRobot and Matrix? Those are decades old.

  • amluto 10 minutes ago

    IMO the most alarming thing currently going on with AI is the huge RL runs that give models some incentive to compete with each other and rather strong incentives to hack things, break rules and otherwise cheat. And the “cyber” initiatives are remarkably examples of doing most of this deliberately.

    Of course, it’s the “frontier labs” doing almost all of this. No one is about to SFT a model that turns into Skynet on its own.

  • amatecha 38 minutes ago

    I find myself thinking the phrase "what could possibly go wrong?" very often these days -- this topic being one that I think it about the most frequently :\

  • darquomiahw 23 minutes ago

    I remember a similar horror movie when scientists with good intentions were performing gain of function research on self-replicating nanomachines and some escaped, killing millions and causing massive economic damage. Of course, nobody held them liable and everyone forgot after a few years.

  • eulgro 7 minutes ago

    1. They are all saying AI is a danger to humanity, but no one is willing to give the slightest details about how exactly the threat will materialize...?

    2. Nor are they willing to consider the possible remedies in case the threat materialize (presumably unplugging the server infrastructure that's consuming gigawatts)...?

    3. They all keep working towards advancing AI despite believing that it might end humanity in the near future...?

    I can't even begin to imagine where the threat to humanity lies. A threat to employment maybe, but that's completely different.

  • seemaze 30 minutes ago

    This appears to be a copy-paste wall of other persons social media posts assembled by a jazz historian with an alarmingly high frequency of the term 'honest' in their blog titles..

  • PerilousD 17 minutes ago

    If you are posting a "This be not good" warning on the toxic crap that is "X", then your credibility quotient with me just dropped by aT LEAST 20%, when it comes to my attention via a previously unknown to me, website (honest-broker.com?). You just dropped another 20% - basically you are a coin flip between reality, bs and delusional so...

  • lifestyleguru 10 minutes ago

    2026's Blair Witch Project.

  • mpalmer 30 minutes ago

    Won't fully dismiss the risk, but...

    Hard not to believe that AI providers really just see this positioning as a way to juice the nascent market for AI security products protecting against AI-based threats. Gotta make money coming and going, and if in the process we superficially resemble a company who cares about the effect it has on the world, all the better!

      GolfPopper 2 minutes ago

      I am very, very tired of what I see as the pretense of AGI or anything like it coming from LLMs.

      I think the biggest risk from "AI" is that chasing the delusions spread by Sam Altman (and others - he's at the front, but very far from a sole actor) is going to do vast, possibly irreparable damage to modern human civilization. And I don't mean cognitive damage from LLM use (although that certainly appears to be possible) but the damage from immense misallocation of resources to ultimately non-productive (if not outright destructive) ends.

      Deep down, I don't believe for a moment that these claims are anything but hype. LLMs are spicy auto complete, backed by immense amounts of compute; as with so many aspects of computer science, clever people can get some amazing and (sometimes) productive outputs. But they are not anything like the fictional dreams and nightmares of "AI". I believe such claims are a mix self-deluded projection and deliberate hype by people who still hope to reap immense personal profits from their implied promises to Install Planetary Overlords.

      But if the hype was all real, if every one of these nightmare scenarios being painted was plausible, then there is no excuse whatsoever for not throwing everyone involved in cells with no access to anything Turning-complete, demolishing the related infrastructure, and establishing an international compact to nuke anyone trying to pursue such AGI until the rubble glows in the dark, because they're an existential threat to humanity.

  • system7rocks 29 minutes ago

    Couple this with the fact that many Silicon Valley/tech nerds are in a bit of a bubble, as has existed for a long time... And we are losing more and more news outlets who can critique these giant, strange companies. I am aware other fields of research do cross-disciplinary conversations from time to time, for example have biotech researchers share dialogue with human rights philosophers, religious scholars, etc to think carefully about the purpose and ethics behind work being done and its potential impact. Is that happening within tech?

      buellerbueller 22 minutes ago

      Tech just hires those people for PR reasons, and those people fall victim to that old saw about the difficulty of getting someone to understand something that runs contrary to the source of their paycheck.

  • ath3nd 31 minutes ago

    [dead]