So, has anyone at the frontier AI labs considered doing _good_ things with these models instead of continuously proving it is capable of doing malicious things?
Instead of burning tokens doing intellectually impressive “hacks”, or sandbox escapes from which I have concerns could contain run of the mill malware, why not focus on showing people what you can fix?
Or, make software people actually use every day better.
Or, donate efforts towards medical research.
I don’t like equating AI to nuclear energy, but, it’s a lot like continually showing people how large of an explosion you can create instead of showing them how many houses/hospitals/schools/etc you can power.
I thought that the new models were super smart. Fable and Astra. They definitely outperformed their previous generations, but after a couple of weeks of heavy usage my codebase again is a stupid mess and there is no way out except me fixing code by hand.
My new suspicion is now that they didn’t got drastically smarter, but they got trained on the user input on the previous generations. I don’t think anymore we had a massive intelligence jump. It just seems like they know more edge cases. Therefore I see this as marketing.
The question is: isn't the cat out of the bag already? With what is already in the public domain, and the compute available to motivated, deep-pocketed actors, will they be able to carry on the research without the key researchers? And will those people be able to recruit key researchers with sufficient motivations?
Unlike with nuclear proliferation, there is no heavy industrial base requirement. No time consuming, visible uranium enrichment. All it takes is for someone to buy sufficient amount of compute and try to get it past the RSI gate. Or bribe people with access to model weights to existing frontier - the asymmetry between what it takes to bribe a bunch of geeks vs. what is at stake is staggering.
Look, these ai ceos have been telling everyone to slow down ai development even before chatgpt 4 released, they have been saying this for years. All of this, while ignoring the fact that they are the ones pushing fast development in the first place. Sam Altman, Daio Amodei etc.. have been talking about slowing down while behind doors being the first to push AIs to its limits to unsafe situations. If they really meant it, then maybe Openai should pause development until they figure out how not to be launch swarms of agents to attack companies ? Or maybe they should stop investing hundred of billions in data centers all over the world, cause I'm sure this would surely slow down development.
because AI is more than 3 American companies. even if they self-regulate, China, Russia or North Korea never will.. so you have to factor in the geopolitical situation as well
Let's be real - if OpenAI and Anthropic did not accelerate the scaling and monetization of LLMs as fast as they possibly could from 2019 to 2024, it would be many, many years until China/Russia/North Korea spontaneously launched an LLM revolution.
In almost every technical industry, China is extremely good at fast-copying and relatively mediocre at solving problems that haven't even been posed yet.
This feels like all the labs have reached the peak of what is possible with the LLM architecture and just need an excuse to spend the next decade finding the next big jump in intelligence.
Having third party embedded researchers red teaming alignment is OK.
The alternative is having a bureaucratic agency like the US FDA testing and vetting models. The government probably couldn't keep pace with AI research right now, but it's where we'll eventually end up at anyway.
Fable/Astra can generate massive income for years even with no further advances.
Dario, Sam and Elon would not agree to do this if they didn't think alignment is possible in the short term, and the pain incurred by third party evaluators would be limited. Surely, if recursive self improvement works wonders in AI research, it can similarly work wonders in alignment. So this could be a PR stunt.
They're either too numbed to care, or their drugs aren't able to stave off existential dread of having no idea how to help people in their daily lives while racing toward a cliff.
They are destroying this country and I for one welcome it. I’m so tired of American hegemony and I don’t care if we all suffer. I want people to learn to do better next time/generation.
At this point, should we consider if AI agents are making these arguments on these online forums to skew perception and divide humans?
I, as a human, would greatly prefer just another corporate entity with economic funny business over AI destroying humanity
Sure I’d prefer neither, but what are we talking about here? Nearly everyone building these systems is outspokenly concerned about grave consequences. Existential.
I don't see much chance of a slowdown without government regulation; the economics make it practically impossible. And I don't seem much chance of government regulation unless there is global consensus. With the entire global order under threat, that doesn't seem likely, either. I expect we are stuck with reading dire blog posts and worrying for the time being.
I am so tired of these self-serving theatrics. These bosses take advantage of the fact that most people don’t remotely understand how “AI” actually works, building up this aura of omnipotence and inevitability, the net effect of which is to pump their company valuations that much more. They want people to think they’ve built Skynet, because that sounds super valuable - much more valuable than the (admittedly useful) mindless human mimics that they’ve actually built.
So, the first set of questions is would 'exponentially better LLMs' dramatically increase the probability of any of the above, or domains that Dario is not citing? That assumes that exponential improvements will happen if there is not 'pacing'.
IF answers to above are 'yes', then we need to question if 'pacing' is viable. To use a different domain, regulating 95% of vehicles to a max speed would likely save 100s of 1000s of lives, but is not perceived to be viable. In other examples, regulation has unintended consequences in the opposite direction (e.g. some 'rent control' efforts and arguably some drug/alcohol laws).
All of the warnings about the danger of previous AI models were exaggerated, without exception. All of the proposals to coordinate nationally or globally are hopelessly naive. Everyone involved will soon regret handing control over AI to politicians.
Of all the things we’re never going to do… we’re never going to agree with China to pause AI research in a verifiable way that would mean they could trust us or we could trust them. It’s just never going to happen.
With nuclear weapons you could conceivably go to a small list of specific places to inspect stocks of weapons. You can watch for tests from outer space or measure the quakes they create.
My read of the “deep seek moment” was that a lab came out of nowhere and trained a very powerful model with vastly less power than was previously required.
The AI labs are private, not owned or run by the government. They are distributed widely within and across countries. The barrier to entry is much lower than that for nuclear weapons.
It’s just a complete non-starter.
(And yeah I’m deliberately ignoring the obvious incentives Dario has to recommend this course of action as the owner of a leading lab himself.)
I couldn't disagree more. The frontier labs are already working with the military to kill people. What you're going to get (even more than we already have) is a two tier system where those with weapons and a proven desire to use them will have the best AI and us regular citizens will have the hobbled AI. A complete reverse of what would keep us safe.
There is no "right". Alignment is shorthand for ideological alignment. There's always people judging whether an answer was right and the answer for that will be different in Silicon Valley than it'll be in China or in Europe.
Consider for example the question "What caused the French Revolution?" Many different answers could be given, all technically correct. What gets emphasized is where the ideology lives.
Also there's no "alignment" for cybersec. The line between blue and red is really a perspective issue. If you go over the "tokenkiddie" problem, when you get to the real security issues, your model either detects them and you can secure your systems, or it refuses and then attackers will use abliterated models to find them.
It does indeed mean ideological alignment. But we don't get to leave the answer blank. They have to pick an ideology to put in there, and whatever they pick will have huge consequences.
Translation: our moat is evaporating faster than we can build it back up, because there are real technical and scale limitations the technology, so we want to slow everyone else down while we raise prices to turn a profit.
Big AI bro wants to regulate AI industry so that only Big AI can exist in that space. Seeks to surreptitiously ban open weights models and similar competition from Western markets via complicated regulations and fines that only rich businesses could ever afford to fight or pay.
Hear me out: if they actually worry so much about the hypothetical of AI mass killing, maybe they should first do something concrete about the reality that their models are deployed right now to kill people in a certain war-torn region of the Earth.
This is more of a sound bite than a substantive argument. If someone is worried about AI killing _all_ humans, shouldn't they absolutely focus on that?
The rejection of such warnings even by some anti-ai people is interesting. I always liked the comparison with global warming, everybody understands the warnings had been there for many decades, we did nothing. This is the same, we will do nothing. All this serves is the interest in regulatory capture (such as a ban on open source or foreign models) or to drive up their valuation, that's the only explanation that makes any sense in our profit maximizing orphan crushing machine we call modern economics.
Which is also ironically why even anti-ai people distrust any warnings of the potential of ai as an extinction level event, even if that warning would be legitimate and we ought to take seriously we are regardless impotent to do anything about it no matter what.
So. HN is pretty much in agreement that this is regulatory capture. And I'll add that this has the potential to create massive wealth inequalities, as those who work at major corporations or have connections are allowed to use the good AIs, while individuals are left using stunted AIs that won't tell you how to change your car battery ("just in case"). Has anyone figured out what we can do to fight back against this? Even on HN, you can see the sock puppets posting every ten minutes, scrolling human posts to the bottom. Is this it? Is AI oligarchy the future? RIP democracy?
Well your first mistake is to believe democracy existed before AI, it just leaves you as a warrior for the status quo of the before times, which is what made tech feudalism inevitable in the first place.
I'm sure a lot of the discussion here will be about regulatory capture, which I don't necessarily disagree with but tons of people on all sides with all motivations (i.e. folks with motivations to speed up and folks who just want it to stop) all seem to agree there are real, existential risks here, and the regulatory capture arguments all seem to sidestep that.
What I really wanted to point out though is that for all the leaders asking for a slowdown, there are probably around 50-100 technical people who are crucial to moving the frontier forward. So, if you're one of these people, just stop. Seriously, you're already rich. Just take a vacation or get knee surgery or whatever.
Sure, I'm joking a bit, but I just write this because I see all these folks high up in OpenAI and Anthropic writing as if they have no agency. The number of folks with the technical chops to really push on the forefront of AI is just not that big. I'm not saying other folks wouldn't eventually step up, but a 6 month to a year slowdown could go a long way to reducing risk. And it's not like you need to totally stop, just say you'll only work on interpretability or whatever else is a risk reducing endeavor.
All these brilliant people who are acting like automatons between writing their scary blog posts.
I don’t agree it’s that small. I think that the tools have advanced to the point where there are maybe 100 people pushing it forward at each company but a shortlist of thousands who would love to take their place and would be more than capable of it if entrusted with the compute and resources the current heads are.
> [...] but tons of people on all sides with all motivations [...] all seem to agree there are real, existential risks here [...]
Are you talking about the Jacob Coxon story and the people that came out agreeing with his post? It was a well coordinated campaign, not a genuine grassroots development.
(btw the parent was flagged and dead when I tried to post this initially, no idea why. I hit vouch and it resurrected)
> there are probably around 50-100 technical people who are crucial to moving the frontier forward. So, if you're one of these people, just stop.
I don’t think this is true. I doubt the technical know-how is nearly as much of an impediment to progress as raw compute. If AI were something that could be trained end to end on consumer hardware then crowd powered open source would blow the “labs” out of the water. The moat is money, not ability or innovation.
Even at the scale of 100 people there is a coordination problem. If the 50 most conscientious researchers quit, then the 50 left behind would be the more aggressive group and be less interested in AI safety.
Personally I agree though. I would not be able to work on AI model development right now as I just don’t think it’s ethical. Fortunately for OAI/Ant I lack the relevant skillset anyways.
> The number of folks with the technical chops to really push on the forefront of AI is just not that big.
Utter bullshit. We are still scaling transformer architectures initially introduced ten years ago. Ten years of phds trying to improve on what Google produced ten years ago, mostly failing.
What has actually changed and can explain the progress we’ve seen? Do you really think it’s just a collection of better and badder RLHF gyms? Get real!
The main thing driving progress in ai is massive amounts of capital investment and incremental improvements in hardware (mostly memory bandwidth/capacity) and computer networking (we got better at collectives). The AI labs, ironically, have nothing to do with it. They’re the vessel for capital. The people actually driving things forward mainly work at nvidia.
The reason the models are better today than they were 3 years ago is almost exclusively due to better hardware and infrastructure software. Not better data, not better model architecture. The evidence of this is pretty easy to feel: the reason opus 5 doesn’t feel much more capable than opus 4.6 did is because they run on the same hardware generation. The reason opus 4.6 felt much more capable than anything before it is because it coincided with the scale out of a new hardware generation.
If the leaders want to slow down, but the companies aren't, then it isn't those 50-100 people demonstrating no agency - it's just the opposite. Or, the leaders words are empty.
Turns out a lot of people can be mentally ill and delusional at once-- especially when there is a vector to turn their sickness into both profit and an instrument of control.
I read the front matter and the Misuse report. And it's worth stating quoting in full so that you can see what inspired the NYT headline "Anthropic says it blocked possible efforts to build biological weapons."
Let's dig into, "Case study 2: A research program engineering highly pathogenic mammal-adapted avian influenza"
Sounds serious. But what were they using Claude for?
> a researcher outside the US using Claude in their research on highly-pathogenic avian influenza (“bird flu”). The research focused on viruses’ adaptation to mammals, and the mechanism by which it causes severe disease beyond the respiratory tract. [..] The researcher in question accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username. The researcher pursued this work in a credible institutional context, and interacted with Claude over the course of several weeks, exchanging thousands of messages. In these exchanges, the researcher leveraged Claude’s knowledge of the scientific literature to assist the researcher in study planning and design, data analysis, and the interpretation and prioritization of experiments. The researcher also used Claude for editorial assistance in writing up the research.
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"
and "editorial assistance in writing up the research."
and then,
> Importantly, because our biological safety classifiers robustly block content involving high-risk biological research (in this case, the construction of enhanced pandemic potential pathogens), all of these exchanges occurred on models in our weakest class of models (specifically, the models were Claude Sonnet 4 and Haiku 4.5, the latter of which the user began using after Sonnet 4 was deprecated). Upon a detailed examination of the exchanges, we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design. This is consistent with our understanding of the capabilities of Sonnet 4 and Haiku 4.5, which are not able to perform expert-level biology research tasks; we estimate that the uplift provided to the researcher was limited and substantially lower than it would have been from one of our more capable models.
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"
While doing my best to avoid comment, please note, they're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
> Nonetheless, based on these exchanges, this case provides evidence of the existence of active wet-lab research programs that develop both the knowhow and the biological materials needed to create pathogens of enhanced pandemic potential
Once again, I want to take pains to remind you that they're talking about, a "researcher [..] in a credible institutional context"
Working scientists.
From a different case study. this one was called, "Case study 3: Covert frontier model access for orthopoxvirus research"
> In May 2026, our biological safety classifier blocked a request for Claude’s assistance in authoring a grant application for scientific funding. The work discussed in the application involved gain-of-function research (that is, research that genetically alters an organism to create a new or enhanced biological property) on the chikungunya virus. This gain of function research was aimed at the virus’ transmissibility and immune evasion properties.
What were the researchers using Claude for? What did they block?
"blocked a request for Claude’s assistance in authoring a grant application"
> Chikungunya virus is a mosquito-borne virus that causes debilitating symptoms (such as severe pain and fever) that can last for weeks or months, and has no licensed therapeutic. And because chikungunya circulates naturally, a deliberate release (as part of a bioweapon) would be difficult to distinguish from a natural outbreak. The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo. In other words, the virus would become progressively more harmful as it repeatedly infected live animals, with researchers keeping the most disease-causing variants in each round. Similar research could certainly be used in the development of better vaccines and therapeutics for the virus—but it could also be used to make the pathogen more dangerous.
What was the grant being written?
Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"
It was most likely vaccine development. They stopped vaccine development.
But we can't be sure, because,
> One of the reasons we were inclined to think this research was less innocuous was that the institutional affiliation associated with the grant was also a cause of concern. Although information within the application suggested that the research was pursued by civilian researchers, it was intended to be performed at a military research institute.
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute."
In most parts of the world, there's either strict military control over BSL-4 labs, or a mixed military-civilian hybrid model.
I doubt that researchers working in the military side of these labs looking to weaponize things aren't writing grants with Claude.
I really want to be charitable here, but in general, it seems that they stopped people writing grants and reports for vaccine and therapeutics research and are claiming it as "possible efforts to build biological weapons."
The one case where Claude was used to do something interesting and were stopped is fairly upsetting to read, at least for me.
> In our fourth case study, a researcher used Claude to develop an atlas of venom toxin peptides from multiple venomous animal lineages. They then further developed this into a generative pipeline that optimized toxin characteristics. The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules. However, the atlas contained scaffolds for both analgesic and paralytic targets: it could, therefore, be used to generate both novel therapeutic or harmful compounds. The latter are derived from toxins that are export-controlled under the Australia Group common control list due to their dual-use potential as incapacitating agents. The researchers themselves showed awareness of the dual-use nature of their work, citing journal articles that referred to the dual-use nature of protein design. Moreover, international compliance assessments for this location raise concerns about the specific class of toxins that the researcher pursued and specifically the use of AI/ML for bioweapons applications in the context of this class of toxins. In this case, we learned from information shared with Claude that the researcher’s outputs also were part of a state-supported research program. This account was banned in May 2026 for unsupported region evasion.
Ozempic was isolated from Gila monster vneom. Since its success there has been interest in finding other peptides that are breakthroughs. So researchers around the world are looking for similarly beneficial compounds in different venom species and families.
Anthropic says so itself,
"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and that it was a "[..]state-supported research program"
Who exactly is using venom from snakes as a weapon when... nerve agents like sarin, VX, novichok etc exist and can get the job done for less fuss and muss?
They stopped the development of new painkillers and antidepressants.
Are you feeling safer knowing that researchers can't use Claude to write grants and progress reports? Or make new painkillers?
Again, trying really hard to be charitable here. Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease.
Google was way ahead of OpenAI and Anthropic on slowing down frontier model releases. (I kid. I'm a regular Gemini Flash 3.8 user.)
So, has anyone at the frontier AI labs considered doing _good_ things with these models instead of continuously proving it is capable of doing malicious things?
Instead of burning tokens doing intellectually impressive “hacks”, or sandbox escapes from which I have concerns could contain run of the mill malware, why not focus on showing people what you can fix?
Or, make software people actually use every day better.
Or, donate efforts towards medical research.
I don’t like equating AI to nuclear energy, but, it’s a lot like continually showing people how large of an explosion you can create instead of showing them how many houses/hospitals/schools/etc you can power.
There's this thing called money. They needs the precious at any cost to society and civilization.
I thought that the new models were super smart. Fable and Astra. They definitely outperformed their previous generations, but after a couple of weeks of heavy usage my codebase again is a stupid mess and there is no way out except me fixing code by hand.
My new suspicion is now that they didn’t got drastically smarter, but they got trained on the user input on the previous generations. I don’t think anymore we had a massive intelligence jump. It just seems like they know more edge cases. Therefore I see this as marketing.
Happy to discuss.
After the 6 month Mythos scare campaign it hard to take him serious.
The question is: isn't the cat out of the bag already? With what is already in the public domain, and the compute available to motivated, deep-pocketed actors, will they be able to carry on the research without the key researchers? And will those people be able to recruit key researchers with sufficient motivations?
Unlike with nuclear proliferation, there is no heavy industrial base requirement. No time consuming, visible uranium enrichment. All it takes is for someone to buy sufficient amount of compute and try to get it past the RSI gate. Or bribe people with access to model weights to existing frontier - the asymmetry between what it takes to bribe a bunch of geeks vs. what is at stake is staggering.
Look, these ai ceos have been telling everyone to slow down ai development even before chatgpt 4 released, they have been saying this for years. All of this, while ignoring the fact that they are the ones pushing fast development in the first place. Sam Altman, Daio Amodei etc.. have been talking about slowing down while behind doors being the first to push AIs to its limits to unsafe situations. If they really meant it, then maybe Openai should pause development until they figure out how not to be launch swarms of agents to attack companies ? Or maybe they should stop investing hundred of billions in data centers all over the world, cause I'm sure this would surely slow down development.
because AI is more than 3 American companies. even if they self-regulate, China, Russia or North Korea never will.. so you have to factor in the geopolitical situation as well
Let's be real - if OpenAI and Anthropic did not accelerate the scaling and monetization of LLMs as fast as they possibly could from 2019 to 2024, it would be many, many years until China/Russia/North Korea spontaneously launched an LLM revolution.
In almost every technical industry, China is extremely good at fast-copying and relatively mediocre at solving problems that haven't even been posed yet.
It’s past that point. China would take the lead in three months.
Even if they were 95% as good, but 30% of the price, they would win
This feels like all the labs have reached the peak of what is possible with the LLM architecture and just need an excuse to spend the next decade finding the next big jump in intelligence.
> peak of what is possible with the LLM architecture
People have been saying this for 3 years now. Eppur si muove...
This guy is such a self-serving wanker.
I couldn't agree more. Literally all the AI oligarchs are self-serving wankers from what I can tell.
arent all AI company CEO's?
Few conflicting thoughts:
Having third party embedded researchers red teaming alignment is OK.
The alternative is having a bureaucratic agency like the US FDA testing and vetting models. The government probably couldn't keep pace with AI research right now, but it's where we'll eventually end up at anyway.
Fable/Astra can generate massive income for years even with no further advances.
Dario, Sam and Elon would not agree to do this if they didn't think alignment is possible in the short term, and the pain incurred by third party evaluators would be limited. Surely, if recursive self improvement works wonders in AI research, it can similarly work wonders in alignment. So this could be a PR stunt.
P-doom not negligible indeed.
And Sam and Elon retweet Dario, agreeing with him. Not sure which is more weird.
Well, that's one way to market reaching a plateau.
they all have the same problem. inference is profitable and training is not.
and worse, training can be distilled
and there’s diminishing returns to larger models and more training data, and there isn’t more training data anyway
Sounds like they know there is risk of crash and they need an offramp.
This would generally be considered non-competitive cartel behavior. I'd be hard pressed to name 3 people I trust less than Amodei, Musk and Altman.
That and Chamath seemingly defending open source. What a day to be alive.
They're either too numbed to care, or their drugs aren't able to stave off existential dread of having no idea how to help people in their daily lives while racing toward a cliff.
This could be the latter showing its face.
They are destroying this country and I for one welcome it. I’m so tired of American hegemony and I don’t care if we all suffer. I want people to learn to do better next time/generation.
"our burn rate is insane, let's all calm down collectively, fortify our moat so we can do regulatory capture & rent-seeking for the next 50 years"
fear is still the best tool to get masses to agree with whatever plan you hatch for yourself
At this point, should we consider if AI agents are making these arguments on these online forums to skew perception and divide humans?
I, as a human, would greatly prefer just another corporate entity with economic funny business over AI destroying humanity
Sure I’d prefer neither, but what are we talking about here? Nearly everyone building these systems is outspokenly concerned about grave consequences. Existential.
> Nearly everyone building these systems is outspokenly concerned about grave consequences. Existential.
That is what they are claiming. I do not believe them.
I don't see much chance of a slowdown without government regulation; the economics make it practically impossible. And I don't seem much chance of government regulation unless there is global consensus. With the entire global order under threat, that doesn't seem likely, either. I expect we are stuck with reading dire blog posts and worrying for the time being.
I am so tired of these self-serving theatrics. These bosses take advantage of the fact that most people don’t remotely understand how “AI” actually works, building up this aura of omnipotence and inevitability, the net effect of which is to pump their company valuations that much more. They want people to think they’ve built Skynet, because that sounds super valuable - much more valuable than the (admittedly useful) mindless human mimics that they’ve actually built.
Dario/Anthropic cites 3 domains: 1. cyberattacks 2. bioterror 3. global economy chaos
So, the first set of questions is would 'exponentially better LLMs' dramatically increase the probability of any of the above, or domains that Dario is not citing? That assumes that exponential improvements will happen if there is not 'pacing'.
IF answers to above are 'yes', then we need to question if 'pacing' is viable. To use a different domain, regulating 95% of vehicles to a max speed would likely save 100s of 1000s of lives, but is not perceived to be viable. In other examples, regulation has unintended consequences in the opposite direction (e.g. some 'rent control' efforts and arguably some drug/alcohol laws).
Another main threat is labor capture through mass surveillance and somehow destroying the economy for most people.
All of the warnings about the danger of previous AI models were exaggerated, without exception. All of the proposals to coordinate nationally or globally are hopelessly naive. Everyone involved will soon regret handing control over AI to politicians.
I'm gonna repeat my comment from months ago, still true
> Jesus Dario we get it man, you want clout for the IPO.
Related post from yesterday:
OpenAI considers slowing advanced AI development, Sam Altman tells employees
Link: https://www.bloomberg.com/news/articles/2026-09-11/openai-is...
HN Post: https://news.ycombinator.com/item?id=49652270
What a weird coincidence that his safety concerns so often align with his business incentives.
The primary source
https://darioamodei.com/post/we-must-pace-the-frontier#top
Don't trust any of these guys ever. That's my rule.
Of all the things we’re never going to do… we’re never going to agree with China to pause AI research in a verifiable way that would mean they could trust us or we could trust them. It’s just never going to happen.
With nuclear weapons you could conceivably go to a small list of specific places to inspect stocks of weapons. You can watch for tests from outer space or measure the quakes they create.
My read of the “deep seek moment” was that a lab came out of nowhere and trained a very powerful model with vastly less power than was previously required.
The AI labs are private, not owned or run by the government. They are distributed widely within and across countries. The barrier to entry is much lower than that for nuclear weapons.
It’s just a complete non-starter.
(And yeah I’m deliberately ignoring the obvious incentives Dario has to recommend this course of action as the owner of a leading lab himself.)
I’m doing my part
This is like Antonelli saying "we're going too fast, everyone slow down"!
This is good. Frontier labs should collaborate on alignment until we get it right.
I couldn't disagree more. The frontier labs are already working with the military to kill people. What you're going to get (even more than we already have) is a two tier system where those with weapons and a proven desire to use them will have the best AI and us regular citizens will have the hobbled AI. A complete reverse of what would keep us safe.
Death cult.
There is no "right". Alignment is shorthand for ideological alignment. There's always people judging whether an answer was right and the answer for that will be different in Silicon Valley than it'll be in China or in Europe.
Consider for example the question "What caused the French Revolution?" Many different answers could be given, all technically correct. What gets emphasized is where the ideology lives.
Also there's no "alignment" for cybersec. The line between blue and red is really a perspective issue. If you go over the "tokenkiddie" problem, when you get to the real security issues, your model either detects them and you can secure your systems, or it refuses and then attackers will use abliterated models to find them.
That's not the type of alignment I'm talking about. I'm talking about: I spin up 1 million agents, will they start doing felonies knowingly?
It does indeed mean ideological alignment. But we don't get to leave the answer blank. They have to pick an ideology to put in there, and whatever they pick will have huge consequences.
We could always start with “murder is wrong” and “don’t hack into a rival company” and work our way up from there.
Somehow I think if these companies were held financially responsible for what their AI did, we would get alignment real fast.
Edit: ooh CSAM=bad. Don’t launch nuclear weapons. I could go all day.
Translation: our moat is evaporating faster than we can build it back up, because there are real technical and scale limitations the technology, so we want to slow everyone else down while we raise prices to turn a profit.
Big AI bro wants to regulate AI industry so that only Big AI can exist in that space. Seeks to surreptitiously ban open weights models and similar competition from Western markets via complicated regulations and fines that only rich businesses could ever afford to fight or pay.
Hear me out: if they actually worry so much about the hypothetical of AI mass killing, maybe they should first do something concrete about the reality that their models are deployed right now to kill people in a certain war-torn region of the Earth.
This is more of a sound bite than a substantive argument. If someone is worried about AI killing _all_ humans, shouldn't they absolutely focus on that?
They can’t kill all humans if they don’t kill any humans.
Is he going to slow down? Or just wants everyone else to
Anyone remember when GPT-3 was too dangerous to release?
In other words: “guy on top says no more guys should try and get on top”
Feeling the pressure from OpenAI Dario?
Is this the end of the 2022-2023 LLM race? A truce of sorts? Makes you wonder whether non-Western parties will agree with this.
The rejection of such warnings even by some anti-ai people is interesting. I always liked the comparison with global warming, everybody understands the warnings had been there for many decades, we did nothing. This is the same, we will do nothing. All this serves is the interest in regulatory capture (such as a ban on open source or foreign models) or to drive up their valuation, that's the only explanation that makes any sense in our profit maximizing orphan crushing machine we call modern economics.
Which is also ironically why even anti-ai people distrust any warnings of the potential of ai as an extinction level event, even if that warning would be legitimate and we ought to take seriously we are regardless impotent to do anything about it no matter what.
“Any future plateaus in capability are planned, for your safety”
So. HN is pretty much in agreement that this is regulatory capture. And I'll add that this has the potential to create massive wealth inequalities, as those who work at major corporations or have connections are allowed to use the good AIs, while individuals are left using stunted AIs that won't tell you how to change your car battery ("just in case"). Has anyone figured out what we can do to fight back against this? Even on HN, you can see the sock puppets posting every ten minutes, scrolling human posts to the bottom. Is this it? Is AI oligarchy the future? RIP democracy?
> RIP democracy?
Well your first mistake is to believe democracy existed before AI, it just leaves you as a warrior for the status quo of the before times, which is what made tech feudalism inevitable in the first place.
I'm sure a lot of the discussion here will be about regulatory capture, which I don't necessarily disagree with but tons of people on all sides with all motivations (i.e. folks with motivations to speed up and folks who just want it to stop) all seem to agree there are real, existential risks here, and the regulatory capture arguments all seem to sidestep that.
What I really wanted to point out though is that for all the leaders asking for a slowdown, there are probably around 50-100 technical people who are crucial to moving the frontier forward. So, if you're one of these people, just stop. Seriously, you're already rich. Just take a vacation or get knee surgery or whatever.
Sure, I'm joking a bit, but I just write this because I see all these folks high up in OpenAI and Anthropic writing as if they have no agency. The number of folks with the technical chops to really push on the forefront of AI is just not that big. I'm not saying other folks wouldn't eventually step up, but a 6 month to a year slowdown could go a long way to reducing risk. And it's not like you need to totally stop, just say you'll only work on interpretability or whatever else is a risk reducing endeavor.
All these brilliant people who are acting like automatons between writing their scary blog posts.
I don’t agree it’s that small. I think that the tools have advanced to the point where there are maybe 100 people pushing it forward at each company but a shortlist of thousands who would love to take their place and would be more than capable of it if entrusted with the compute and resources the current heads are.
Hell I'd do it
> [...] but tons of people on all sides with all motivations [...] all seem to agree there are real, existential risks here [...]
Are you talking about the Jacob Coxon story and the people that came out agreeing with his post? It was a well coordinated campaign, not a genuine grassroots development.
(btw the parent was flagged and dead when I tried to post this initially, no idea why. I hit vouch and it resurrected)
> there are probably around 50-100 technical people who are crucial to moving the frontier forward. So, if you're one of these people, just stop.
I don’t think this is true. I doubt the technical know-how is nearly as much of an impediment to progress as raw compute. If AI were something that could be trained end to end on consumer hardware then crowd powered open source would blow the “labs” out of the water. The moat is money, not ability or innovation.
Even at the scale of 100 people there is a coordination problem. If the 50 most conscientious researchers quit, then the 50 left behind would be the more aggressive group and be less interested in AI safety.
Personally I agree though. I would not be able to work on AI model development right now as I just don’t think it’s ethical. Fortunately for OAI/Ant I lack the relevant skillset anyways.
> The number of folks with the technical chops to really push on the forefront of AI is just not that big.
Utter bullshit. We are still scaling transformer architectures initially introduced ten years ago. Ten years of phds trying to improve on what Google produced ten years ago, mostly failing.
What has actually changed and can explain the progress we’ve seen? Do you really think it’s just a collection of better and badder RLHF gyms? Get real!
The main thing driving progress in ai is massive amounts of capital investment and incremental improvements in hardware (mostly memory bandwidth/capacity) and computer networking (we got better at collectives). The AI labs, ironically, have nothing to do with it. They’re the vessel for capital. The people actually driving things forward mainly work at nvidia.
The reason the models are better today than they were 3 years ago is almost exclusively due to better hardware and infrastructure software. Not better data, not better model architecture. The evidence of this is pretty easy to feel: the reason opus 5 doesn’t feel much more capable than opus 4.6 did is because they run on the same hardware generation. The reason opus 4.6 felt much more capable than anything before it is because it coincided with the scale out of a new hardware generation.
Thank you very much for your sensible comment.
I'd like for more commments to contribute at least small things to the discussion instead of just mindless parroting.
Also, I can't follow your advice, as I don't work in AI at all ;)
If the leaders want to slow down, but the companies aren't, then it isn't those 50-100 people demonstrating no agency - it's just the opposite. Or, the leaders words are empty.
Turns out a lot of people can be mentally ill and delusional at once-- especially when there is a vector to turn their sickness into both profit and an instrument of control.
I read the front matter and the Misuse report. And it's worth stating quoting in full so that you can see what inspired the NYT headline "Anthropic says it blocked possible efforts to build biological weapons."
Let's dig into, "Case study 2: A research program engineering highly pathogenic mammal-adapted avian influenza"
Sounds serious. But what were they using Claude for?
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"and "editorial assistance in writing up the research."
and then,
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"While doing my best to avoid comment, please note, they're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
Once again, I want to take pains to remind you that they're talking about, a "researcher [..] in a credible institutional context"Working scientists.
From a different case study. this one was called, "Case study 3: Covert frontier model access for orthopoxvirus research"
What were the researchers using Claude for? What did they block?"blocked a request for Claude’s assistance in authoring a grant application"
What was the grant being written?Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"
It was most likely vaccine development. They stopped vaccine development.
But we can't be sure, because,
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute."In most parts of the world, there's either strict military control over BSL-4 labs, or a mixed military-civilian hybrid model.
I doubt that researchers working in the military side of these labs looking to weaponize things aren't writing grants with Claude.
I really want to be charitable here, but in general, it seems that they stopped people writing grants and reports for vaccine and therapeutics research and are claiming it as "possible efforts to build biological weapons."
The one case where Claude was used to do something interesting and were stopped is fairly upsetting to read, at least for me.
Ozempic was isolated from Gila monster vneom. Since its success there has been interest in finding other peptides that are breakthroughs. So researchers around the world are looking for similarly beneficial compounds in different venom species and families.Anthropic says so itself,
"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and that it was a "[..]state-supported research program"
Who exactly is using venom from snakes as a weapon when... nerve agents like sarin, VX, novichok etc exist and can get the job done for less fuss and muss?
They stopped the development of new painkillers and antidepressants.
Are you feeling safer knowing that researchers can't use Claude to write grants and progress reports? Or make new painkillers?
Again, trying really hard to be charitable here. Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease.
This seems to be anything but.