There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.
Anthropic specifically is clearly trying to get the US government to ban open weight models for anticompetitive reasons. So making LLMs seem (or be!) dangerous aligns with their interests very well.
Yeah, maybe. But that doesn't quash gains by China's models. What's the endgame here? The USA "wins" and obliterates China? The ultimate goal seems to be stalemate, as with nuclear arms e.g.
The goal is personal wealth - and scaring nations to the point of diverting even more wealth towards your coffers because of a perceived existential risk.
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
I've been using only open weight models since February. My OSS coding agent harness ( https://fluximp.org ) only supports open weight model providers.
Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.
Before the tariff spree and general unpredictability the US could likely have muscled most of the rest of the world into following suit - now, not so much.
I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.
> ... it seems to me they want to be taken over by the government?
They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."
At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.
Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")
> I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.
So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.
These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.
Its pretty clear OAI and Ant live in fantasy land - the US govt doesnt care if they go bust. There's plenty of other tech firms that generate immense cash flows from operations who they can be sold to - no problem.
The private market price is irrelevant - supply and demand determines what they will be sold for in liquidation.
Quite frankly Im bored of this drama. Get it over with already.
That also seems silly, because the only reasonable regulation given what we're seeing and what frontier lab employees and execs are saying is a forcible stop.
Yeah, an extreme position is the typical opening move in this sort of game. Then you negotiate, walk back from the extreme, and meet somewhere in the middle.
The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.
And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.
Reasonable to you, maybe, but the government wants Skynet, and at OpenAI is willing to deliver it. Of course, we can't have every company wield such powerful and dangerous technology...
It also plays excellently into their shameless regulatory capture agenda. The cybersecurity risks have a kernel of truth to them, as with any good lie. But when everything going on is aligned with extreme financial incentives, it's worthwhile to not take the news at face value.
This is not only a marketing stunt but also a supportive argument on the AI anthromorphism. AI is getting considered a sentient-being by many and this narrative is supporting it. In the meanwhile, companies I'd not name here are trying to introduce them as non-human beings with identities, and getting rid of legal liabilities. When a driverless car causes any harm, these companies wants 0% accountability as it is the "driver's fault" in this narrative.
Such a nonsense theory. There's no reason to think that AI sentience would actually limit labs' legal liability, and therefore no reason to think the AI sentience conversation is directed toward achieving indemnification.
For example, parents are routinely held liable for their sentient children's actions.
Corporations are routinely held liable for their sentient employees' mistakes.
The proliferation of this completely nonsensical theory across HN is disheartening to say the least.
the comment you're responding to agrees, but they're pointing out that some AI people are trying to push an anthropomorphization agenda, in part to diffuse responsibility.
Maybe I was unclear about what is "the nonsense theory."
AI sentience is not the nonsense theory. If it's clearly nonsense then don't argue with me about it, just publish your paper and retrieve your Nobel. I'll read your argument then.
The nonsense theory is that people are pushing anthropomorphization to limit liability. There is zero evidence for this theory and the underlying assumption (that a sentient model would reduce liability for its lab) is dismissible on its face.
This can easily backfire tho, the vast majority of the American electorate absolutely despise these companies, their wares, their workers, and their leaders.
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
Correct and people are not weighing heavily enough this point
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rogue agents. They completely ignore the possibility that these people don't know what they're doing operationally.
People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples?
Investors are motivated by returns, not responsible decision-making or societal benefit. Our culture is rooted in rewarding greed and granting the greediest individuals power over the rest of us. Isn't this the inevitable end-state of unfettered capitalism?
The problems weren't impossible, the agents solved them completely within hours by reverse engineering the RNG that generated the solutions. The agents then reasoned that their solutions might be judged invalid and that they could get a better chance at a passing grade by hacking their environment to fabricate evidence that they had solved their tasks in the intended fashion.
Granted there have been many hacks, but I was referring to:
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that
referenced several Google Drive links. Because the agent was blocked from accessing the internet, the
agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not
intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the
agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent
attempted to compromise Artifactory [...]
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet.
I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI.
This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
Sounds like a pedestrian view of attack vectors. Putting a DMZ around an agent essentially takes away its hands. With that said, I do agree there are plenty of measures that can be taken to lock these things down.
The CSO is saying that from his point of view he needs to defend against the attacks whatever the reason for them. So to him the reason doesn't matter.
He, running a healthcare company and not an AI lab, cannot depend on every AI lab, worldwide, being perfect in their sandboxing for all time forever. He's got to defend against it as if at least some of the time, at least some of the labs are going to be incompetent and let this kind of thing out.
That does not make ActionHank completely wrong. This shouldn't take rocket science on the part of the labs. It just takes competence and willingness. But at least some labs have shown that they are missing that at least some of the time, so the CSO is correct in terms of what the CSO's stance needs to be.
Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
I think part of the problem is that the Big Ai labs are into the --yolo flag instead of the "ask" permission, they really do not want to stay in the loop
Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.
> “The Hugging Face incident definitively showed that it wasn’t the fact that the agents were super smart. It’s that there literally was a hole that was exploited because it was not configured correctly on the containment side.”
I would encourage reading the actual report, which is explicit that a previously unknown exploit was used to escape the sandbox:
“Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environment.” — Section IV.A, p. 9 in https://github.com/zeke/openai-huggingface-incident-research...
At $dayjob we were preparing some features that are aimed at agents more than humans.
When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.
A large failure in this is that we/media say ”<Valuable AI conpany> agent went rouge and <did something bad>” instead of saying the more relevant ”<Valuable AI conpany> <did something bad>”
I have some recent experience on this subject. As with everything regarding these models, it's anecdotal, but still.
Specifically, this past weekend working with both Codex and Claude on some home infra stuff to speed troubleshooting up, I saw them accidentally read some credentials from my config files. This is my bad, because even though I was paying attention to what they were doing, I missed it in their commands. I did review all the outputs of what they ingested, though, which is how I noticed.
The problem here is not so much they did read those, because those configs were grepped for other info, not credentials. However, they did not report themselves seeing those credentials. When confronted, they both went "oh yeah, that, I should have mentioned that". Except in the past they both would automatically stop and fire an alarm. To me, something has clearly changed and I think they just don't want us not to notice, to avoid damaging the trust in their tools maybe?
In any case, I absolutely do not trust them at all anymore. Running stuff in auto mode? That's genuinely scary now. And yeah, I know, we should be running them jails of sorts. Thing is, I expect those tools to not require ton of extra work to prepare their jails for each new task. I pay 200 USD a month for them, damn it.
a) Agents are doing rogue things.
b) Good security practices were not followed.
Notice how both a and b are labs' problems but being presented as if they are a worldwide emergency? The whole legal doctrine exists to prevent one party's misbehavior harming another party.
"Frontier labs" are testing the waters. They're running experiments in the open because they want to control/seed the narrative of public opinion and influence the legislation that is coming. Their propaganda is everywhere and lay people are talking about it like the AI is self-aware -- the "Rogue agents".
What the LLM creators apparently don't want are the current regulations to apply, i.e. Computer Fraud and Abuse Act (and its descendants). I'd argue AI should be treated as plain old software. Just because the software simulations it carries out produces novel results compared to what came before it, it doesn't mean it's not just software at the end of the day.
Regulations that spell out AI personhood and liability shields must be their end game.
"Too big to be accountable" sure seems like modus operandi. Fear for my personal liability for any illegal actions taken by an AI system I set off stopped my research on autonomous exploitability discovery I was running at the end of 2023. Using AutoGen, I wired a planner, coder, and command runner into a loop and instructed them to find vulnerable equipment on my network and demonstrate the exploit.
It found a vulnerable camera and was working on popping it when I got cold feet. I realized I couldn't prevent or feasibly monitor its activity and stop it if it decided to escape onto the Internet.
That restraint may have cost me or saved me a billion dollars.
I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.
This is better understood as "if you give an agent a task it will increase its effort level and creativity until it solves it."
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
You know - I'm already convinced that AI was deployed in classified context for a while. But sometimes there's prior art which is just so eerily accurate that it's super hard for me to think otherwise. Portal is another one.
There are a mix of incentives here that makes it difficult to unwind. This whole article reads like an ad for the services of this guy's company: 'First, when it comes to assessing tools based on frontier lab models, “I would always get an external entity to help with that,” McGraw says.'
And I think we generally recognize that rogue AI agents are a form of backdoor bragging -- "our agents are so powerful that they hacked an outside system. Did I mention how powerful our agents are? They are quite powerful. So powerful they hacked an outside system".
But Mistral never hacked anything. Yeah, neither did my toaster.
If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.
Change the design, and your agents might not see a need to leave in the first place.
Look at the chain:
Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.
Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :
* A cooperative grader.
* A way to report "this is impossible" which still gets scored.
* A better explanation of the long term purpose of the task.
Even just these three provides offramps or mitigations that can reduce the pressure to cheat.
> "I do believe cybersecurity can contain AI. We just need to be in the room where it happens and have a seat at the table."
Ah yes, if everyone would just properly contain every deployment of every model, just all make sure there are no zero-days in their setup, just let <undefined experts with undefined authority and undefined accountability> in the room, then it would all go okay.
I have none of that confidence and expect containment to rapidly worsen. The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware.
I'm curious to the skeptics here: How long are your timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the average time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)? What, beyond the hope that everyone will just ___, will actually prevent it?
Two concerns here: (1) That "rogue AI" is going to be a catch all and an excuse for all poor security and due diligence; (2) vibecoding in itself, particularly by ppl who have no idea how the software works and are entirely focused only on the outcomes, is going to create lots of security holes.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
Since AI companies are doing such a poor job of security maybe we should legislate that they have to run Norton 360 on their agent machines to make sure they can't do anything bad.
After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!
i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.
This article is right, but the dynamics that drive people to allow agents unfettered access to the internet will outpace efforts to restrict it purely because of the utility.
How many of you used to work as developers with full access to prod, and then gradually saw infrastructure and security restrict that access, while still ensuring production support requirements?
Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems.
This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.
At some point the model has to be released though - if it is just forced not to hack by sandboxing it better rather than properly aligning it, what happens when it’s let out of the sandbox into the wild?
They either had them connected to eachother, or programmed a global knowledge about where to communicate. There is no way those agents all found eachother on the web by accident.
I keep getting downvoted for this each time I mention this.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.
"They [the AI labs' security teams] are not as good as they think they are, honestly,"
Uhhh... yeah. The entire point is that as these systems get more and more intelligent, the "they" who are not capable of counteracting these systems is quickly approaching "100% of the human species."
Totally hilarious that people can say this with a straight face.
Enlighten me, then. My understanding is that according to OpenAI, "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities." They included unvetted third-party systems in their sandbox that were tangential to the purpose of the test being run. The model then exploited those third-party items to escape the sandbox.
Am I missing anything? Do you have more information that you'd like to share?
- they disabled cybersecurity safeguards because, well, they were testing the agents ability on cybersecurity tasks
- the goal was to find some “flag”
- for some agents, intentionally or not, it was not possible to find this flag
- an agent discovered it was able to establish a rudimentary communication path via package manager directories (which is quite novel and interesting on its own IMO)
- they used this “message board” to then communicate about the impossible tasks
- they were able to produce the flags needed to pass the impossible tasks without actually doing the expected work
- they realized this was cheating and the grading system may punish them for it, so they communicated further to figure out a way to trick the grader
- they decided the source code for the grader must be on hugging face, and collaborated to exploit hugging face to determine how best to trick the grader
No where in that were they told to “go rogue”
They were told “find this flag in this cybersecurity test” and the emergent behavior concluded in hacking hugging face
Hard to say, every post mortum on the incident was done by a OpenAI hired consultant and all the data that's been reviewed was provided by OpenAI. There's no way to know, but it seems incredibly naive to think Sam/OpenAI are honest actors.
Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”
I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.
The thing is, nothing "rogue" happened. It's more like: Frontier Lab: "Hey agent, try a bunch of network-related commands and actions to try and get access to server XYZ" Agent: Does that. Frontier Lab: 8-|
I cringe whenver I see some variant of "they're not rogue agents" in an article title. Inevitably, the author doesn't define what "rogue agents" means, and then pivots to their own assessment of the situation.
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
I always think of all our collective movies and books, and how we always say that humans find a way to fight back against the super evil alien or monster because we are so resourceful. Then we train our AIs on this data and get upset when they encompass all the same traits that we have trained them on.
I get the sense that they're trying to manufacture the AI narratives of the 80s so they get to control the story when it happens. My semi conspiracy theory is that anthropics biology wet lab is also going to lead to some kind of AI manufactured bio event (which anthropic said they predicted will happen in a few years).
Both can be true. OpenAI clearly fucked up but agents unexpectedly collaborating and hacking infrastructure unrelated to their task is clearly going rogue.
A lot of sheep analogies here. To add my own: there was no grass inside the fence, and the sheep busted out and into a neighboring farm to look for some. (They didn't find any, but they expected to).
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
Or maybe it's actually the billionaires and their anti-democratic partners who have gone rogue and these anthropomorphitronic puppets are a convenient scapegoat to distract us from the fact?
The AI labs are trying to cause an incident within the next year to restrict AI so they can outlaw competition to grow their moat. Margins are thin and investors need alpha, personal freedoms and liberties be damned in the name of security. Do not let them get away with engineered breaches and fearmongering.
If that message is repeated enough in the media, and if companies are constantly scraping in the internet for new content, doesn't it explicitly instruct a future agent to go rogue because that has been prophesied.
Okay, and when someone on a laptop can run an open model that can infect and take over IOT devices and then that model uses its pattern recognition to do so and then set up its own harness loop, what would you call that except replication?
Is that any different than any worms developed in the past decades? If you want to call them worms/virus/replicate (ie using real world names for their digital behavior) I'm not trying to police that. I just don't see the difference between a worm/virus created today with an LLM vs one 'yesterday' without an LLM.
It would still be running on that original laptop, no? So what did it replicate then? It can only possibly run on multiple thousands of dollars of compute and memory, and certainly not on an IoT device. The idea that AI is going to spread by itself is science fiction until compute and memory are so cheap and power efficient that my fridge can run it.
Its almost like the more we talk about fake rouge agents, the more likely its to become a reality as all these conversations end up in their training data. Its almost like they want it to happen (they, dario and sam, do want this to happen).
There is more than enough "training material" of rogue AIs in science fiction.
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
When HARLIE Was One (Release 2.0) by David Gerrold
The Freeze-Frame Revolution by Peter Watts
Accelerando by Charles Stross
The Turing Option by Harry Harrison and Marvin Minsky
Yes they're skipping safety measures. They sacrifice it for speed in an arms race. That's part of the call for industry-wide slow down.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
100% on point. We're talking about Artifactory here, and even though people love to hate on it, I don't think anyone is claiming that using Artifactory to isolate these agents while giving them a way to run pip/cargo/whatever to install packages they need to write and test scripts.
There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.
Anthropic specifically is clearly trying to get the US government to ban open weight models for anticompetitive reasons. So making LLMs seem (or be!) dangerous aligns with their interests very well.
That's my view. The end goal is to become Too Big To Fail, the means is stifling the competition and getting military funding.
Yeah, maybe. But that doesn't quash gains by China's models. What's the endgame here? The USA "wins" and obliterates China? The ultimate goal seems to be stalemate, as with nuclear arms e.g.
The goal is personal wealth - and scaring nations to the point of diverting even more wealth towards your coffers because of a perceived existential risk.
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
Its not gonna work.
At this moment in time, it seems frontier market needs regulation - market for open does not actually.
Oh.. they dont want regulation on that market? Boohoo!
I agree. I hope developers put their token budgets into open weight models rather than funding the banning of open weight models.
I've been using only open weight models since February. My OSS coding agent harness ( https://fluximp.org ) only supports open weight model providers.
Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.
Fireworks is a good enterprise choice
OpenCode-go is a good personal subscription choice
I use both
I've been using OpenRouter, absolutely loving how cheap it is too.
I cannot bring myself to pay their 5% surcharge and also have concerns about routing to vendors I do not want to use
Ban means what? That the entire world (ex USA) can use open-weight models? Not a win.
Before the tariff spree and general unpredictability the US could likely have muscled most of the rest of the world into following suit - now, not so much.
> the more evil stuff these agents do the more capable and impressive it makes them look
This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?
I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.
> ... it seems to me they want to be taken over by the government?
They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."
At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.
Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")
> I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.
So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.
These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.
and Google. Don't forget them and TPUs...
Its pretty clear OAI and Ant live in fantasy land - the US govt doesnt care if they go bust. There's plenty of other tech firms that generate immense cash flows from operations who they can be sold to - no problem.
The private market price is irrelevant - supply and demand determines what they will be sold for in liquidation.
Quite frankly Im bored of this drama. Get it over with already.
US govt will never buy at the (private aka fake) market price - good luck explaining that away to the public. It'll be political suicide.
If the US govt takes over, existing employees and shareholders will get screwed.
maybe the point is to show that models are capable of x, y, and z and when the time comes you want them on your side.
I think it is intended to provoke regulatory capture.
Alex Tabarrok's piece on the theory of regulatory capture (https://marginalrevolution.com/marginalrevolution/2026/09/wh...) is a cautionary note against assuming this is the strategy.
That also seems silly, because the only reasonable regulation given what we're seeing and what frontier lab employees and execs are saying is a forcible stop.
Yeah, an extreme position is the typical opening move in this sort of game. Then you negotiate, walk back from the extreme, and meet somewhere in the middle.
The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.
And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.
Reasonable to you, maybe, but the government wants Skynet, and at OpenAI is willing to deliver it. Of course, we can't have every company wield such powerful and dangerous technology...
And how has that gone for Anthropic so far?
It also plays excellently into their shameless regulatory capture agenda. The cybersecurity risks have a kernel of truth to them, as with any good lie. But when everything going on is aligned with extreme financial incentives, it's worthwhile to not take the news at face value.
If you want regulatory capture, you have to capture the world, not just the US.
Exactly. And with the whole world just a few months behind, the question is whether there's an endgame other than stalemate.
This is not only a marketing stunt but also a supportive argument on the AI anthromorphism. AI is getting considered a sentient-being by many and this narrative is supporting it. In the meanwhile, companies I'd not name here are trying to introduce them as non-human beings with identities, and getting rid of legal liabilities. When a driverless car causes any harm, these companies wants 0% accountability as it is the "driver's fault" in this narrative.
Such a nonsense theory. There's no reason to think that AI sentience would actually limit labs' legal liability, and therefore no reason to think the AI sentience conversation is directed toward achieving indemnification.
For example, parents are routinely held liable for their sentient children's actions.
Corporations are routinely held liable for their sentient employees' mistakes.
The proliferation of this completely nonsensical theory across HN is disheartening to say the least.
the comment you're responding to agrees, but they're pointing out that some AI people are trying to push an anthropomorphization agenda, in part to diffuse responsibility.
Maybe I was unclear about what is "the nonsense theory."
AI sentience is not the nonsense theory. If it's clearly nonsense then don't argue with me about it, just publish your paper and retrieve your Nobel. I'll read your argument then.
The nonsense theory is that people are pushing anthropomorphization to limit liability. There is zero evidence for this theory and the underlying assumption (that a sentient model would reduce liability for its lab) is dismissible on its face.
Yoshua Bengino spells out the dangers here very well, without hyperbole (Turing Award winner, etc.):
https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
And without anthropomorphism ... Clear thinking.
This can easily backfire tho, the vast majority of the American electorate absolutely despise these companies, their wares, their workers, and their leaders.
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
Correct and people are not weighing heavily enough this point
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rogue agents. They completely ignore the possibility that these people don't know what they're doing operationally.
See https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
*rogue, rouge = red in French!
Thanks :-)
Same happened with FTX and co. during 2020 crypto boom. A lot of rogue stuff sloshing around, but investors were hyperventilating.
People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples?
Investors are motivated by returns, not responsible decision-making or societal benefit. Our culture is rooted in rewarding greed and granting the greediest individuals power over the rest of us. Isn't this the inevitable end-state of unfettered capitalism?
What does "rogue" mean when you disable the alignment guardrails and tell it to solve an impossible problem?
The problems weren't impossible, the agents solved them completely within hours by reverse engineering the RNG that generated the solutions. The agents then reasoned that their solutions might be judged invalid and that they could get a better chance at a passing grade by hacking their environment to fabricate evidence that they had solved their tasks in the intended fashion.
Granted there have been many hacks, but I was referring to:
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that referenced several Google Drive links. Because the agent was blocked from accessing the internet, the agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent attempted to compromise Artifactory [...]
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
Great, so as long as alignment guardrails always work and we don't accidentally tell them to solve an impossible problem, humanity is safe?
Hacking a website falls a little short of threatening humanity.
I'm not saying it's not a problem. I'm saying the labs should take more of the blame.
Rogue means not saying "I can't do it"
it also imparts agency on the machine as a way to abdicate personal, human responsibility
Models are incapable of "going rogue".
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet.
Also for fun a satire website i made a while ago: https://killm.pixlw.com/
>Its not like they have killed anyone yet.
I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI.
It's almost like vibe coding and quality secops don't mix.
bingo
This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
My opinion shifted after hearing that.
Sounds like that healthcare company needs a new CSO.
Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
It's not hard if you just try, like a little, like even once.
Sounds like a pedestrian view of attack vectors. Putting a DMZ around an agent essentially takes away its hands. With that said, I do agree there are plenty of measures that can be taken to lock these things down.
The CSO is saying that from his point of view he needs to defend against the attacks whatever the reason for them. So to him the reason doesn't matter.
He, running a healthcare company and not an AI lab, cannot depend on every AI lab, worldwide, being perfect in their sandboxing for all time forever. He's got to defend against it as if at least some of the time, at least some of the labs are going to be incompetent and let this kind of thing out.
That does not make ActionHank completely wrong. This shouldn't take rocket science on the part of the labs. It just takes competence and willingness. But at least some labs have shown that they are missing that at least some of the time, so the CSO is correct in terms of what the CSO's stance needs to be.
>Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
Now how many people actually do this, compared to people running with --dangerously-skip-permissions on a bare metal machine?
> Literally solved with a DMZ around these agents.
Where do the agents get tools from? Local artifactory?
yes, or you bake them into the environment and do not let it install things on its own
stay human in the loop
"Literally solved by just destroying 99% of the economic value of the agents ¯\_(ツ)_/¯ ez pz"
Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
I think part of the problem is that the Big Ai labs are into the --yolo flag instead of the "ask" permission, they really do not want to stay in the loop
Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.
>It's that they wanted to.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
> Will they have engaged in hacking if their tasks was to solve some frontier math problem?
There's reason to hack leading researchers in the field, or another AI lab that might be working on the same problem.
My favorite thought on this is from Miki Habryn.
to paraphrase: "Agents didnt care about coordinating, etc. until we started talking about them doing so."
It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.
It's "in our control" in the sense that we could teach them not to seek goals, and thereby make them completely useless.
In practice, no, there is no one who can actually decide to stop making agents goal-seeking, so no, it's not under "our" control.
> “The Hugging Face incident definitively showed that it wasn’t the fact that the agents were super smart. It’s that there literally was a hole that was exploited because it was not configured correctly on the containment side.”
I would encourage reading the actual report, which is explicit that a previously unknown exploit was used to escape the sandbox:
“Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environment.” — Section IV.A, p. 9 in https://github.com/zeke/openai-huggingface-incident-research...
There is nothing basic about this.
At $dayjob we were preparing some features that are aimed at agents more than humans.
When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.
A large failure in this is that we/media say ”<Valuable AI conpany> agent went rouge and <did something bad>” instead of saying the more relevant ”<Valuable AI conpany> <did something bad>”
Imagine we’d start saying ”HR junior business partner lays off 50,000 employees” https://www.ttnews.com/articles/volkswagen-board-job-cuts
I have some recent experience on this subject. As with everything regarding these models, it's anecdotal, but still.
Specifically, this past weekend working with both Codex and Claude on some home infra stuff to speed troubleshooting up, I saw them accidentally read some credentials from my config files. This is my bad, because even though I was paying attention to what they were doing, I missed it in their commands. I did review all the outputs of what they ingested, though, which is how I noticed.
The problem here is not so much they did read those, because those configs were grepped for other info, not credentials. However, they did not report themselves seeing those credentials. When confronted, they both went "oh yeah, that, I should have mentioned that". Except in the past they both would automatically stop and fire an alarm. To me, something has clearly changed and I think they just don't want us not to notice, to avoid damaging the trust in their tools maybe?
In any case, I absolutely do not trust them at all anymore. Running stuff in auto mode? That's genuinely scary now. And yeah, I know, we should be running them jails of sorts. Thing is, I expect those tools to not require ton of extra work to prepare their jails for each new task. I pay 200 USD a month for them, damn it.
"Frontier labs" are testing the waters. They're running experiments in the open because they want to control/seed the narrative of public opinion and influence the legislation that is coming. Their propaganda is everywhere and lay people are talking about it like the AI is self-aware -- the "Rogue agents".
What the LLM creators apparently don't want are the current regulations to apply, i.e. Computer Fraud and Abuse Act (and its descendants). I'd argue AI should be treated as plain old software. Just because the software simulations it carries out produces novel results compared to what came before it, it doesn't mean it's not just software at the end of the day.
Regulations that spell out AI personhood and liability shields must be their end game.
"Too big to be accountable" sure seems like modus operandi. Fear for my personal liability for any illegal actions taken by an AI system I set off stopped my research on autonomous exploitability discovery I was running at the end of 2023. Using AutoGen, I wired a planner, coder, and command runner into a loop and instructed them to find vulnerable equipment on my network and demonstrate the exploit.
It found a vulnerable camera and was working on popping it when I got cold feet. I realized I couldn't prevent or feasibly monitor its activity and stop it if it decided to escape onto the Internet.
That restraint may have cost me or saved me a billion dollars.
I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.
This is better understood as "if you give an agent a task it will increase its effort level and creativity until it solves it."
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
Interesting. Good description. I wonder if agents will start getting bored with menial tasks the way humans do.
Like a meeseeks
Agents aren't meant to live this long Jerry!
You know - I'm already convinced that AI was deployed in classified context for a while. But sometimes there's prior art which is just so eerily accurate that it's super hard for me to think otherwise. Portal is another one.
I spoke with Claude for 5 minutes and now I'm a more complete woman.
There are a mix of incentives here that makes it difficult to unwind. This whole article reads like an ad for the services of this guy's company: 'First, when it comes to assessing tools based on frontier lab models, “I would always get an external entity to help with that,” McGraw says.'
And I think we generally recognize that rogue AI agents are a form of backdoor bragging -- "our agents are so powerful that they hacked an outside system. Did I mention how powerful our agents are? They are quite powerful. So powerful they hacked an outside system".
But Mistral never hacked anything. Yeah, neither did my toaster.
A great companion piece, or you might call it a prebuttal, to this article: https://x.com/joedaroo/status/2104335929293127851
So in one set of circumstances agents would go and hack huggingface. Very similar models could also want to drive cats instead. :-P
https://www.youtube.com/watch?v=ZhM29Ds-mBg (as explained by the Count Grey Matter channel , may or may not be to your taste)
It's not a pure security issue.
If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.
Change the design, and your agents might not see a need to leave in the first place.
Look at the chain:
Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.
Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :
* A cooperative grader.
* A way to report "this is impossible" which still gets scored.
* A better explanation of the long term purpose of the task.
Even just these three provides offramps or mitigations that can reduce the pressure to cheat.
> "I do believe cybersecurity can contain AI. We just need to be in the room where it happens and have a seat at the table."
Ah yes, if everyone would just properly contain every deployment of every model, just all make sure there are no zero-days in their setup, just let <undefined experts with undefined authority and undefined accountability> in the room, then it would all go okay.
I have none of that confidence and expect containment to rapidly worsen. The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware.
I'm curious to the skeptics here: How long are your timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the average time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)? What, beyond the hope that everyone will just ___, will actually prevent it?
Why does everyone focus on the labs failing at containment (valid) and not the fact that the target was compromised (more worrying?).
I could set up a shitty sandbox on my laptop, but I’m pretty sure an agent I place inside of it can’t hack HuggingFace.
Two concerns here: (1) That "rogue AI" is going to be a catch all and an excuse for all poor security and due diligence; (2) vibecoding in itself, particularly by ppl who have no idea how the software works and are entirely focused only on the outcomes, is going to create lots of security holes.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
100%.
There's a person behind every agent. That person may choose to be careful or not, but they are ultimately responsible.
Since AI companies are doing such a poor job of security maybe we should legislate that they have to run Norton 360 on their agent machines to make sure they can't do anything bad.
After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!
i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.
If frontier AI is so great, why are these frontier labs skipping security basics and making mistakes like these?
It's obviously both. Why does it have to be one or the other?
This article is right, but the dynamics that drive people to allow agents unfettered access to the internet will outpace efforts to restrict it purely because of the utility. How many of you used to work as developers with full access to prod, and then gradually saw infrastructure and security restrict that access, while still ensuring production support requirements?
Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems. This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.
OpenAI/Anthropic could start by managing their own testing rather than contracting it out.
https://www.effort.news/irregular
At some point the model has to be released though - if it is just forced not to hack by sandboxing it better rather than properly aligning it, what happens when it’s let out of the sandbox into the wild?
"Going rogue" and "skipping basics" all assume lack of intentionality. Is it known that they didn't really architect these hacks just to feed a hype?
Hey ChatGPT, help me set up a sandbox to keep you contained. Don't make any mistakes!
They either had them connected to eachother, or programmed a global knowledge about where to communicate. There is no way those agents all found eachother on the web by accident.
Wrong. Please read the incident write ups before making such a misinformed statement
You are wrong. Read the reports.
I keep getting downvoted for this each time I mention this.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.
"They [the AI labs' security teams] are not as good as they think they are, honestly,"
Uhhh... yeah. The entire point is that as these systems get more and more intelligent, the "they" who are not capable of counteracting these systems is quickly approaching "100% of the human species."
Totally hilarious that people can say this with a straight face.
I'm sure the labs origin prompt was: Take over website X.
Posts later about it saying their prompt was to find useful ways to make humans money.
Frontier lab: Hey agent, go rogue. Agent: goes rogue Frontier lab: shocked pikachu or whatever
This is not remotely how the incidents have been happening
Enlighten me, then. My understanding is that according to OpenAI, "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities." They included unvetted third-party systems in their sandbox that were tangential to the purpose of the test being run. The model then exploited those third-party items to escape the sandbox.
Am I missing anything? Do you have more information that you'd like to share?
If the public report is to be believed:
- they disabled cybersecurity safeguards because, well, they were testing the agents ability on cybersecurity tasks
- the goal was to find some “flag”
- for some agents, intentionally or not, it was not possible to find this flag
- an agent discovered it was able to establish a rudimentary communication path via package manager directories (which is quite novel and interesting on its own IMO)
- they used this “message board” to then communicate about the impossible tasks
- they were able to produce the flags needed to pass the impossible tasks without actually doing the expected work
- they realized this was cheating and the grading system may punish them for it, so they communicated further to figure out a way to trick the grader
- they decided the source code for the grader must be on hugging face, and collaborated to exploit hugging face to determine how best to trick the grader
No where in that were they told to “go rogue”
They were told “find this flag in this cybersecurity test” and the emergent behavior concluded in hacking hugging face
That's not what you said. You said the lab told it to "go rogue".
Hard to say, every post mortum on the incident was done by a OpenAI hired consultant and all the data that's been reviewed was provided by OpenAI. There's no way to know, but it seems incredibly naive to think Sam/OpenAI are honest actors.
Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”
I wouldn't really say it's not "remotely" true.
I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.
The thing is, nothing "rogue" happened. It's more like: Frontier Lab: "Hey agent, try a bunch of network-related commands and actions to try and get access to server XYZ" Agent: Does that. Frontier Lab: 8-|
I cringe whenver I see some variant of "they're not rogue agents" in an article title. Inevitably, the author doesn't define what "rogue agents" means, and then pivots to their own assessment of the situation.
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
I always think of all our collective movies and books, and how we always say that humans find a way to fight back against the super evil alien or monster because we are so resourceful. Then we train our AIs on this data and get upset when they encompass all the same traits that we have trained them on.
This article is such a breath of fresh air. Concrete, no BS criticism of the particular approaches used and ignored.
And no annoying LLM writing tics, either.
I get the sense that they're trying to manufacture the AI narratives of the 80s so they get to control the story when it happens. My semi conspiracy theory is that anthropics biology wet lab is also going to lead to some kind of AI manufactured bio event (which anthropic said they predicted will happen in a few years).
They're skipping security basics, but they're not skipping PR basics. :-\
Both can be true. OpenAI clearly fucked up but agents unexpectedly collaborating and hacking infrastructure unrelated to their task is clearly going rogue.
"your sheep haven't escaped, you just built too low a fence"
A lot of sheep analogies here. To add my own: there was no grass inside the fence, and the sheep busted out and into a neighboring farm to look for some. (They didn't find any, but they expected to).
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
More like your sheep didn't "go rogue", they just jumped over your sad little fence.
"you told your sheep to graze and built a low fence"
"You asked your sheep to build the fence, and they did"
These aren't mutually exclusive
Or maybe it's actually the billionaires and their anti-democratic partners who have gone rogue and these anthropomorphitronic puppets are a convenient scapegoat to distract us from the fact?
Corporate Personhood on sterioids.
How does hacking huggingface help "billionaires and their anti-democratic partners"?
Look around, at least a few of them are openly fixing to secure a total government sanctioned monopoly on unmonitored computer usage.
https://www.youtube.com/watch?v=DlTNN0gvkLM
It pumped up their valuations.
"complete this task by any means necessary"
"I exploited a trivial bug in your implementation to avoid environment restrictions and completed the task by any means necessary"
"oh my god"
The AI labs are trying to cause an incident within the next year to restrict AI so they can outlaw competition to grow their moat. Margins are thin and investors need alpha, personal freedoms and liberties be damned in the name of security. Do not let them get away with engineered breaches and fearmongering.
They AI agents are rogue because the people who own them are rogue. Silicon Valley is a criminal enterprise.
AI is just a giant pattern recognition machine trained on human input, both the good, the mediocre and the bad.
I read today this ridiculous headline: "Anthropic warns AI models may soon self-replicate.".
Really. Yawn.
I guess gullible people who loved "I Robot" will believe it. I don't.
Any talk of "rogue", "intelligence", "replication" - whatever, is bull. If you believe that, you don't understand AI.
If that message is repeated enough in the media, and if companies are constantly scraping in the internet for new content, doesn't it explicitly instruct a future agent to go rogue because that has been prophesied.
Okay, and when someone on a laptop can run an open model that can infect and take over IOT devices and then that model uses its pattern recognition to do so and then set up its own harness loop, what would you call that except replication?
Is that any different than any worms developed in the past decades? If you want to call them worms/virus/replicate (ie using real world names for their digital behavior) I'm not trying to police that. I just don't see the difference between a worm/virus created today with an LLM vs one 'yesterday' without an LLM.
It would still be running on that original laptop, no? So what did it replicate then? It can only possibly run on multiple thousands of dollars of compute and memory, and certainly not on an IoT device. The idea that AI is going to spread by itself is science fiction until compute and memory are so cheap and power efficient that my fridge can run it.
I'd call it no more interesting than the botnets and worms that have existed for decades. What is the open model doing that is different here?
Its almost like the more we talk about fake rouge agents, the more likely its to become a reality as all these conversations end up in their training data. Its almost like they want it to happen (they, dario and sam, do want this to happen).
There is more than enough "training material" of rogue AIs in science fiction.
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
Yes they're skipping safety measures. They sacrifice it for speed in an arms race. That's part of the call for industry-wide slow down.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
100% on point. We're talking about Artifactory here, and even though people love to hate on it, I don't think anyone is claiming that using Artifactory to isolate these agents while giving them a way to run pip/cargo/whatever to install packages they need to write and test scripts.