I'll use this opportunity while it's still early to share my experience with Argon.
I got it via the "Pro" plan, exclusively for personal task usage. So no agentic programming.
For starters, the limits are dreadfully low. Five messages on medium is already enough to trigger a 4 hour cooldown. Secondly, the best part is the integration with Google search and Maps. Makes it seamless for trip planning. Lastly it's the only "SOTA" model in a long time that has actively hallucinated details. I don't know if it's because the Gemini harness is not asking it to verify information, but I've had some real Deepseek V3-like lies in responses.
Either way, I find it difficult to get invested in Gemini because nobody will have access to Carbon for a few months anyways.
Yeah... Given that everyone else seems to make models available either immediately or within days of announcement, idk why Google feels it's competitive or acceptable to do these 'available at some point' releases.
Capabilities are also improving so fast that this fomo doesn't really keep people excited. If I can't run it, I'll sorta just click off. And given how they handled the Argon release, I've not even bothered look at this Carbon one. Fully expected it to not be usable yet.
I similarly had Argon respond with a vast hallucination. It surprised me because it wasn't a little wrong on the details but a fully false verbose response that completely fabricated fabricated everything. I'd recently had Claude Code help me create transcripts for some audio notes from my mother in law and decided to see how some other models would handle the same task sans harness. Argon made up a long conversation from a police sting operation with under cover operatives seemingly just because it read the file names and identified that they would have come from a recording device.
Honestly the fiction writing was pretty impressive!
Suddenly very curious what the llm roleplay people think about this one lol. Fortunately I've already learned to stifle my curiosity on this specific subject.
hmm, I haven't tried Carbon yet. But I have been using Argon on and off via agent-shell - https://github.com/xenodium/agent-shell and unofficial ACP client.
I am also using it via Google pro tier and did not immediately run into quota issues. But as someone else said in previous discussion of this - Google Pro tier already includes lot of stuff (including Youtube premium etc), the AI plans are just a nice addon.
Agreed on the limits, they're quite frustrating. I've still been preferring Argon to Opus 5.5 though, because even though Argon gets details wrong, I find that it's much more creative and willing to explore ideas. In my experience Opus tends to be rather condescending and won't make room for less orthodox ideas to develop. For what I'm doing, I'd much rather have good ideas that I implement mostly by hand rather than regressing to the mean and have the LLM write the code.
Same. I subscribed to Gemini pro and requested a refund literally the same day. Asked it to put a technical image on a presentation (gave it the image) and it kept generating a different image of a peaceful landscape... Yeah, very pleasant but totally unrelated.
In my experience with antigravity-cli for coding Argon gets the task finished much faster than 3.8 Flash and with a higher quality output. The quotas on Pro are about 2 hours of usage per week for Argon. Its a really nice model to use. Only thing that is annoying is that it still struggles with tool use on the gemini website.
> I find it difficult to get invested in Gemini because nobody will have access to Carbon for a few months anyways.
I feel like you're being overly dramatic here. Argon was 13 - 15 days.
Argon was announced September 30, made available on October 13 to Ultra everywhere, and Pro in the Gemini App. Pro plans got Argon on October 15 in AGY.
This is the first time they've slow-rolled a model. Who knows how long they'll wait on Carbon.
The Ultra-lite plans ($99/month) get some pretty good quota where you can actually use Argon some more. Maybe Google shouldn't be grouping together plans, but it should also be noted that $20/month also gets you 5TB of storage on Drive. If you paid for Dropbox, you get 3TB for $18/month (without any AI model usage).
Since you're a googler, can you trigger some internal process to improve the connection between google services?
Gemini cannot create "my maps". I like creating a map of stuff for travel and now I ask it to create an importable file (which it doesn't always get right) but it should honestly do that itself.
And I don't mean that from within the Gemini app, I'd like a shareable list of places, which you can do via a very outdated maps UI
That's the problem for me. Google can have the world's best model; it doesn't matter to me, because I effectively can't use it.
Both Opus 5.5 and Sol 6.1 are extremely good and provide a lot of inference on their $200 plans, and then there are the Chinese models - DeepSeek 4.1 Flash, MiMo, and StepFun all offer a lot of value per cost.
At the moment, I don't see a niche where it makes sense to use Google's models, regardless of how good they are.
Note that Argon was rolled out for IDEs for people on paid AI Pro on Oct 15/Oct 16 (a couple days after Ultra users), but of course, usage limits are extremely low
I mean this specifically for general tasks, like personal research. Small stuff. I was asking for espresso machine recommendations and it literally came up with a non-existent machine that supposedly met all my requirements. I've never seen that with ChatGPT, Claude or Kimi. It's quite surprising how it did that _after_ doing a Google search.
I am having a very positive experience with Argon: since it was launched I accessed it through Ultragravity and connected via a Google Cloud account. For one month I have been trying to move the needle in SHA256 cryptoanalysis with Claude to not avail and Argon moved "two needles".
Since then I tried it for experimenting on diverse fields. For example, this week there was a discussion here in HN about the non-continuation of Python funding on the JIT, so I "played" again to removing the GIL and moving to the JIT in Python, it did it. I played with many other things, even the Riemann Hypothesis (it just removed their rational parameter rounding only moves 0.875 towards 0.87495702, nothing significant though).
Many other relatively great things were done this week in parallel. I have some budget for research.
I remember when Opus 5.5 came out, we had a discussion about whether it was nerfed a week later [1]. It's not possible to do so with Gemini 4 models since none has been made available the day they are announced.
Without us being able to access Carbon when it is announced, there's no way to know the Carbon we get is the same thing as what has been benchmarked here a few weeks later when it is made available. I think benchmarks should verify what is benchmarked is exactly the same as what's been released.
Also, their tooling still is complete shit. I'll not be forced to use Antigravity thing even if it is available and clearly better than Opus or Astra. Their subscription is useless. API can be used with anything you got but that's gonna cost.
There's an issue with this index where many models are listed in the model chooser, but missing from the index. E.g. Qwen 3.8 Flash Next is listed but missing, as are older Claude (4.6, 4.5, etc) or OpenAI models. This makes it a bit tricky to see the improvements over time.
They removed Qwen Flash Next and GLM Flash from their benchmarks, as of two days ago. They're not publishing any figures at all for them (they show as "N/A").
You know what. When Gemini 3.1 Pro Preview came out it was better than Opus at the time. It was my main model for 3 weeks.
Until it got lobotomised. Then it lost most of it's capability.
Similar situation happened to Anthropic's Fable. It was an amazing model until Opus 5.5 came out.
So now, I'm not even looking at a "new" model until it is a month or two old. By then we will know if they will destroy ability to make it cheaper to host.
Not OP, but I asked Gemini 3.8 Flash this question (via self-hosted LibreChat) and got an excellent answer:
I'm trying to figure out how heat dissipation works. Specifically, I have a block of aluminum with extruded fins (as a heat sink) and I'm trying to understand how many 30-watt LEDs I can place on it. I have a fan as well, and I'd like to understand how fast I need to run it at different power levels (the LEDs are dimmable). I can gather measurements empirically (which I'll do), but I also want to understand the basic mathematical basis I should use for estimating these values.
It kind of doesn’t make sense. Modern google is typically kind of bad at building products and very good at getting them in customer hands. This is the opposite of that.
It does make sense, when you consider that Google had its earnings report yesterday, and missed its targets. $GOOG is already down 13%. It's an announcement purely for their investors.
Carbon scoring below Opus 5.5 on coding highlights how magnificent Opus 5.5 is as a coding agent. With just a $20/month sub, it performs a ton of work at a very high quality.
I tried Argon for a large historical HTR transcription project recently, since I heard it was really good at this. It was really good, but like all top US models I've used it was
* Definitively very good at domain specific knowledge. It can read sloppy 19th century danish gothic handwriting with faded ink just fine. Not perfectly, but probably better than I could have done with a day of struggle for every page.
* Expensive.
* Bad at actually doing what I said. I had told it in the prompt to be careful not to miss text, but I hadn't explicitly told it that some images may have text in double-column layout. So on those images - over half of my 300-page document - it produced an excellent transcription of the first column, and then considered itself done. Want to try again with a more careful prompt? That'll be another $20.
For ocr, you have to practice, practice, practice with your prompts on small sample sizes before one-shotting a book. I create a json schema for the output that describes the structure in detail (using ai from my prose descriptions of structure of course). I have specific tough pages I use on openrouter to test every new model I am interested in. Gemini has and does absolutely dominate for this. But flash can be better because it doesn't overthink things.
I did do test images, but apparently not enough. It's not 100% of images it drops the second column on either. On many it gets both, on some it only gets the second column.
I also found that it ignores other instructions sporadically ("Don't use letter variants or ligatures in the output, use the normal Danish/Norwegian alphabet when at all possible", still a few Unicode tall s'es in the output.)
It's good, but it's so frustrating since "worse" models manage to follow such instructions much better.
Gemini 2.5 flash and 3 flash were highly underrated models that performed way better than the peers and we used the hell out of them.
Since then none of the Google models have competed well with the others at their price points. Long context performance drop, instruction following issues, unnecessarily complex and sloppish outputs.
Argon is no different, it feels just an iteration better than 3.8 flash. I don't trust the AA ranking yet.
Even with gemini 2.5 or so i remember the models used to just simply hang in production and randomly used to collate tokes and give in the output stream vs giving one by one which was weird.
Also the guardrails were implemented in a very bad manner. By default the models used to raise a default guardrail error on basic queries.
I haven’t tried Gemini for coding yet. I did consider buying their subscription, but based on what I’ve seen from other forums, Gemini has never been a frontier in coding, even with good “benchmaxing”.
Just so everyone is aware of the "coincidences" that happen with this index:
- Two days ago AA announced their v5 Index which added a few new benchmarks and tweaked a few things.
- Today the newest model announced from a big lab tops the charts.
I'm not saying that AA is definitely taking money from the big labs and changing the benchmark to make the labs' new models look better, but it kind of looks that way. This pattern of behavior has happened before, where the index gets tweaked right around when a new model comes out, and the new model is on top. AA does not, AFAIK, provide a way to see how the new models would have scored under the older index versions. You could probably calculate it yourself, but I'm not going to do that much legwork just for an HN comment.
It's interesting because they publish the index methodology, so you can't really accuse them of not being transparent, but I don't really believe that the choices to update the index were completely independent of the model they happened to be evaluating at the time.
Related: The DXOMark camera scoring method also had similar behavior, where they would release new testing methodology right before flagship smartphones were released. Again, they are technically being transparent about the process, but it kind of smells like the "independent" testers may not be as independent as the average user would hope.
Yeah, it is very suspicious. Each time they update their ranking algorithm, the closed models from US labs jump up and any previously high ranking open model jumps down. It happened with MiMo, Kimi, GLM. All were right there amongst the very top, and each time that happened, AA followed quickly with an index update that shunted them dramatically lower.
Check out vals.ai for an alternative. Not sure how independent they really are either, but it's at least good for a cross reference.
And how do they realize their current index is broken? They either have some recency bias, or indeed favor the big American labs (more likely, since a new Chinese model topping the charts triggers a shakeup).
It's odd because I wonder if anybody actually cares about such scores. It's not like millions of end users look at the score and use it to make a decision what to buy. Subscriptions can easiliy be canceled and people anyway use the models they "feel" work best for them. Hardly anybody I know uses an even remotely scientific method to evaluate model performance. It's mostly vibes with a little bit of (mostly flawed) A/B testing sprinkled in.
I think part of it is to justify capex. "Yeah, we spent x billions on this, but look, we have the best models, we're beating OpenAI and Anthropic, we're winning, it's all worth it!"
This is just a hunch since I have not used Argon very much, and I have not used it for coding (deepseek 4.1 flash 1017 with dsh, and qwen 3.8 max are so good for coding tasks that local models have difficulty with).
My hunch is that Google will end up being best for non coding tasks. Google can concede coding to other vendors who pay Google for infra.
People always love to think the AI race is a duopoly (OpenAI vs. Anthropic), but from what I have experienced, Gemini was always holding on its own, and I find myself using it more than Claude.
True, I use LLMs as glorified search engines to unearth material to read myself, but Gemini is much more useful if you decide to learn yourself. It also offers interesting expansions to your questions.
While I don't offload my work to LLMs (e.g.: Write this project, I'll be watching TV), I find them useful for closing my knowledge gap by collating information, and Google's Gemini is good at that.
They have an insane amount of resources but are so bloated, have a ton of bureaucracy, and have lost a lot of their top talent. Looks like they have figured things out now though.
Not everything is writing code and search. My uncle works in mortgages and the (very large) firm he works for is entirely Gemini. The way he talks about it is with wonderment in his eyes while he goes into detail on how it went through multiple documents and found unaccounted for cases. I think Google is doing fine in these kinds of industries.
I wouldn't even bet that Google is better than OpenAI and Anthropic for these categories, but they seem to be well used outside of our direct space.
"We're taking a deliberate approach to rollout and are actively engaged in the U.S. government’s voluntary process for pre-release model testing. Starting this week, we will make Carbon available to a subset of cyber defenders and will gradually phase access to additional trusted partners through our Fairwind Program. We’re continuing to gather feedback and iterate on guardrails before making Carbon available to developers, businesses, and consumers via our product surfaces as soon as possible."
I don’t care and in my opinion AA does a disservice to their users by allowing such blatant manipulation. What would happen if every AI company were to include their unreleased models like this? Put the unreleased models on a separate page and leave the headline benchmark rankings for models that you can actually use.
By the time this model is available for public use I'd imagine it won't be #1 anymore, which is pretty sad. Even when Google gets back on the ball they seem to find a way to fuck it up.
It's unobtanium so nobody should care about it. Any frontier AI company can nowadays create models with ridiculous resource and token usage that will get extremely high scores, but they're of no use when you can't use them.
I take this opportunity to give feedback on Argon. The limitations are a problem. I have a Pro subscription and Argon already stops working after 3-4 prompts with long delays afterwards. It's virtually useless right now even when you pay monthly for it. It is better for some tasks than 3.1 Pro, though. Not much in my experience but it can sometimes fix errors that 3.1 has problems with. I never use Flash models since they often give abysmal results.
A secret model I trained last afternoon is the actual #1 on benchmarks.
No, you can't use it. You can't even see it, and I won't provide you any information. But you must trust me that I am the genius who made the best model in the whole world.
Spare me the celebration. Artificial Analysis should not be granting any company the favor of a #1 ranking for a model that isn't released to the public.
Google's model release is pure PR puffery. It's no coincidence that it was released on the day of its earnings call.
I'll use this opportunity while it's still early to share my experience with Argon.
I got it via the "Pro" plan, exclusively for personal task usage. So no agentic programming.
For starters, the limits are dreadfully low. Five messages on medium is already enough to trigger a 4 hour cooldown. Secondly, the best part is the integration with Google search and Maps. Makes it seamless for trip planning. Lastly it's the only "SOTA" model in a long time that has actively hallucinated details. I don't know if it's because the Gemini harness is not asking it to verify information, but I've had some real Deepseek V3-like lies in responses.
Either way, I find it difficult to get invested in Gemini because nobody will have access to Carbon for a few months anyways.
Yeah... Given that everyone else seems to make models available either immediately or within days of announcement, idk why Google feels it's competitive or acceptable to do these 'available at some point' releases.
Capabilities are also improving so fast that this fomo doesn't really keep people excited. If I can't run it, I'll sorta just click off. And given how they handled the Argon release, I've not even bothered look at this Carbon one. Fully expected it to not be usable yet.
I don't think you are their target audience, unless you are in their promotions committee or a large holder of their stock.
The target is to make a celebratory announcement.
I similarly had Argon respond with a vast hallucination. It surprised me because it wasn't a little wrong on the details but a fully false verbose response that completely fabricated fabricated everything. I'd recently had Claude Code help me create transcripts for some audio notes from my mother in law and decided to see how some other models would handle the same task sans harness. Argon made up a long conversation from a police sting operation with under cover operatives seemingly just because it read the file names and identified that they would have come from a recording device.
Honestly the fiction writing was pretty impressive!
Suddenly very curious what the llm roleplay people think about this one lol. Fortunately I've already learned to stifle my curiosity on this specific subject.
hmm, I haven't tried Carbon yet. But I have been using Argon on and off via agent-shell - https://github.com/xenodium/agent-shell and unofficial ACP client.
In my limited usage, it has fixed issues which has not been correctly fixed by sol-6.1 (medium) - https://github.com/auto-gnufied/markdown-modern/commit/a189a...
I am also using it via Google pro tier and did not immediately run into quota issues. But as someone else said in previous discussion of this - Google Pro tier already includes lot of stuff (including Youtube premium etc), the AI plans are just a nice addon.
Agreed on the limits, they're quite frustrating. I've still been preferring Argon to Opus 5.5 though, because even though Argon gets details wrong, I find that it's much more creative and willing to explore ideas. In my experience Opus tends to be rather condescending and won't make room for less orthodox ideas to develop. For what I'm doing, I'd much rather have good ideas that I implement mostly by hand rather than regressing to the mean and have the LLM write the code.
Same. I subscribed to Gemini pro and requested a refund literally the same day. Asked it to put a technical image on a presentation (gave it the image) and it kept generating a different image of a peaceful landscape... Yeah, very pleasant but totally unrelated.
In my experience with antigravity-cli for coding Argon gets the task finished much faster than 3.8 Flash and with a higher quality output. The quotas on Pro are about 2 hours of usage per week for Argon. Its a really nice model to use. Only thing that is annoying is that it still struggles with tool use on the gemini website.
> I find it difficult to get invested in Gemini because nobody will have access to Carbon for a few months anyways.
I feel like you're being overly dramatic here. Argon was 13 - 15 days.
Argon was announced September 30, made available on October 13 to Ultra everywhere, and Pro in the Gemini App. Pro plans got Argon on October 15 in AGY.
This is the first time they've slow-rolled a model. Who knows how long they'll wait on Carbon.
The Ultra-lite plans ($99/month) get some pretty good quota where you can actually use Argon some more. Maybe Google shouldn't be grouping together plans, but it should also be noted that $20/month also gets you 5TB of storage on Drive. If you paid for Dropbox, you get 3TB for $18/month (without any AI model usage).
(googler, opinions are my own)
Since you're a googler, can you trigger some internal process to improve the connection between google services?
Gemini cannot create "my maps". I like creating a map of stuff for travel and now I ask it to create an importable file (which it doesn't always get right) but it should honestly do that itself.
And I don't mean that from within the Gemini app, I'd like a shareable list of places, which you can do via a very outdated maps UI
problem is, i don't care about the extra storage at all.
my whole family of 5 people is using in total 160 GB. personally I use about 15 GB
> For starters, the limits are dreadfully low.
That's the problem for me. Google can have the world's best model; it doesn't matter to me, because I effectively can't use it.
Both Opus 5.5 and Sol 6.1 are extremely good and provide a lot of inference on their $200 plans, and then there are the Chinese models - DeepSeek 4.1 Flash, MiMo, and StepFun all offer a lot of value per cost.
At the moment, I don't see a niche where it makes sense to use Google's models, regardless of how good they are.
I have it on $200 Ultra and it feels like Opus 5.5 (I do not think about limits) rather than gpt-6* models where I constantly hit a wall.
Note that Argon was rolled out for IDEs for people on paid AI Pro on Oct 15/Oct 16 (a couple days after Ultra users), but of course, usage limits are extremely low
The best [whatever] is the one that is the most available.
> Lastly it's the only "SOTA" model in a long time that has actively hallucinated details
Could you expand on this? I’ve never seen a frontier model that doesn’t hallucinate details fairly often, what do you mean by “actively” here?
I mean this specifically for general tasks, like personal research. Small stuff. I was asking for espresso machine recommendations and it literally came up with a non-existent machine that supposedly met all my requirements. I've never seen that with ChatGPT, Claude or Kimi. It's quite surprising how it did that _after_ doing a Google search.
I am having a very positive experience with Argon: since it was launched I accessed it through Ultragravity and connected via a Google Cloud account. For one month I have been trying to move the needle in SHA256 cryptoanalysis with Claude to not avail and Argon moved "two needles".
Since then I tried it for experimenting on diverse fields. For example, this week there was a discussion here in HN about the non-continuation of Python funding on the JIT, so I "played" again to removing the GIL and moving to the JIT in Python, it did it. I played with many other things, even the Riemann Hypothesis (it just removed their rational parameter rounding only moves 0.875 towards 0.87495702, nothing significant though).
Many other relatively great things were done this week in parallel. I have some budget for research.
I remember when Opus 5.5 came out, we had a discussion about whether it was nerfed a week later [1]. It's not possible to do so with Gemini 4 models since none has been made available the day they are announced.
Without us being able to access Carbon when it is announced, there's no way to know the Carbon we get is the same thing as what has been benchmarked here a few weeks later when it is made available. I think benchmarks should verify what is benchmarked is exactly the same as what's been released.
Doesn't mean jack without access to it.
I used to defend Google's models months ago, but switched to Luna for less intelligent models.
But with this bs of not making it even public, pushing off release of newer models late, I lost faith in Google.
Yeah. It is vaporware at this stage.
Also, their tooling still is complete shit. I'll not be forced to use Antigravity thing even if it is available and clearly better than Opus or Astra. Their subscription is useless. API can be used with anything you got but that's gonna cost.
There's an issue with this index where many models are listed in the model chooser, but missing from the index. E.g. Qwen 3.8 Flash Next is listed but missing, as are older Claude (4.6, 4.5, etc) or OpenAI models. This makes it a bit tricky to see the improvements over time.
They removed Qwen Flash Next and GLM Flash from their benchmarks, as of two days ago. They're not publishing any figures at all for them (they show as "N/A").
https://artificialanalysis.ai/models/qwen3-8-flash-next
https://artificialanalysis.ai/models/glm-5-3-flash/
You know what. When Gemini 3.1 Pro Preview came out it was better than Opus at the time. It was my main model for 3 weeks.
Until it got lobotomised. Then it lost most of it's capability.
Similar situation happened to Anthropic's Fable. It was an amazing model until Opus 5.5 came out.
So now, I'm not even looking at a "new" model until it is a month or two old. By then we will know if they will destroy ability to make it cheaper to host.
Gemini, for whatever reason, has consistently been the best for explaining scientific concepts for me.
Especially with the recent addition of graphic visualization widgets
It manages to strike the perfect balance between depth and style of prose that other Frontier models just can't quite seem to match
Do you have some prompt examples I can run. Thanks
I'll publish a few of my convos
1. Sleep debts effect on muscle size and strength gain
https://share.gemini.google/Gvdb5Q02INrl
2. Fish oil and extra cellular vesicles
https://share.gemini.google/fem9EBCe4mfy
3. Pharmacology question about human-equivalent dose
https://share.gemini.google/RWNYJ2eiNjcU
Not OP, but I asked Gemini 3.8 Flash this question (via self-hosted LibreChat) and got an excellent answer:
Good for them, having a frontier model that nobody can actually use.
It kind of doesn’t make sense. Modern google is typically kind of bad at building products and very good at getting them in customer hands. This is the opposite of that.
It does make sense, when you consider that Google had its earnings report yesterday, and missed its targets. $GOOG is already down 13%. It's an announcement purely for their investors.
Carbon scoring below Opus 5.5 on coding highlights how magnificent Opus 5.5 is as a coding agent. With just a $20/month sub, it performs a ton of work at a very high quality.
I tried Argon for a large historical HTR transcription project recently, since I heard it was really good at this. It was really good, but like all top US models I've used it was
* Definitively very good at domain specific knowledge. It can read sloppy 19th century danish gothic handwriting with faded ink just fine. Not perfectly, but probably better than I could have done with a day of struggle for every page.
* Expensive.
* Bad at actually doing what I said. I had told it in the prompt to be careful not to miss text, but I hadn't explicitly told it that some images may have text in double-column layout. So on those images - over half of my 300-page document - it produced an excellent transcription of the first column, and then considered itself done. Want to try again with a more careful prompt? That'll be another $20.
For ocr, you have to practice, practice, practice with your prompts on small sample sizes before one-shotting a book. I create a json schema for the output that describes the structure in detail (using ai from my prose descriptions of structure of course). I have specific tough pages I use on openrouter to test every new model I am interested in. Gemini has and does absolutely dominate for this. But flash can be better because it doesn't overthink things.
I did do test images, but apparently not enough. It's not 100% of images it drops the second column on either. On many it gets both, on some it only gets the second column.
I also found that it ignores other instructions sporadically ("Don't use letter variants or ligatures in the output, use the normal Danish/Norwegian alphabet when at all possible", still a few Unicode tall s'es in the output.)
It's good, but it's so frustrating since "worse" models manage to follow such instructions much better.
> That'll be another $20.
That seems arguably pretty cheap compared to the human alternatives.
I do get super annoyed at how specific you've got to be with these tools. They are like almost like malicious compliance geniuses lol.
Gemini 2.5 flash and 3 flash were highly underrated models that performed way better than the peers and we used the hell out of them.
Since then none of the Google models have competed well with the others at their price points. Long context performance drop, instruction following issues, unnecessarily complex and sloppish outputs.
Argon is no different, it feels just an iteration better than 3.8 flash. I don't trust the AA ranking yet.
Even with gemini 2.5 or so i remember the models used to just simply hang in production and randomly used to collate tokes and give in the output stream vs giving one by one which was weird.
Also the guardrails were implemented in a very bad manner. By default the models used to raise a default guardrail error on basic queries.
250tk/s is really the sweet spot for chat IMO, results from Flash were fast and usually good quality.
I haven’t tried Gemini for coding yet. I did consider buying their subscription, but based on what I’ve seen from other forums, Gemini has never been a frontier in coding, even with good “benchmaxing”.
Let’s see, maybe they’ll surprise this time.
Is this a pro 3.1 situation that by the time ot reached the users it produced no more work than the flash variant or competitor for absurd pricing?
Just so everyone is aware of the "coincidences" that happen with this index:
- Two days ago AA announced their v5 Index which added a few new benchmarks and tweaked a few things.
- Today the newest model announced from a big lab tops the charts.
I'm not saying that AA is definitely taking money from the big labs and changing the benchmark to make the labs' new models look better, but it kind of looks that way. This pattern of behavior has happened before, where the index gets tweaked right around when a new model comes out, and the new model is on top. AA does not, AFAIK, provide a way to see how the new models would have scored under the older index versions. You could probably calculate it yourself, but I'm not going to do that much legwork just for an HN comment.
It's interesting because they publish the index methodology, so you can't really accuse them of not being transparent, but I don't really believe that the choices to update the index were completely independent of the model they happened to be evaluating at the time.
Related: The DXOMark camera scoring method also had similar behavior, where they would release new testing methodology right before flagship smartphones were released. Again, they are technically being transparent about the process, but it kind of smells like the "independent" testers may not be as independent as the average user would hope.
Yeah, it is very suspicious. Each time they update their ranking algorithm, the closed models from US labs jump up and any previously high ranking open model jumps down. It happened with MiMo, Kimi, GLM. All were right there amongst the very top, and each time that happened, AA followed quickly with an index update that shunted them dramatically lower.
Check out vals.ai for an alternative. Not sure how independent they really are either, but it's at least good for a cross reference.
Have you considered the other direction? They test a new model, and realize their current index is broken, so they improve it?
At the previous iterations they were accused of favoring OpenAI. Now they are accused of favoring Google.
And how do they realize their current index is broken? They either have some recency bias, or indeed favor the big American labs (more likely, since a new Chinese model topping the charts triggers a shakeup).
It's odd because I wonder if anybody actually cares about such scores. It's not like millions of end users look at the score and use it to make a decision what to buy. Subscriptions can easiliy be canceled and people anyway use the models they "feel" work best for them. Hardly anybody I know uses an even remotely scientific method to evaluate model performance. It's mostly vibes with a little bit of (mostly flawed) A/B testing sprinkled in.
So ... why bother to cheat?
> So ... why bother to cheat?
I think part of it is to justify capex. "Yeah, we spent x billions on this, but look, we have the best models, we're beating OpenAI and Anthropic, we're winning, it's all worth it!"
From all of the terrible benchmarks of AIs, the AA one is the least terrible.
This is just a hunch since I have not used Argon very much, and I have not used it for coding (deepseek 4.1 flash 1017 with dsh, and qwen 3.8 max are so good for coding tasks that local models have difficulty with).
My hunch is that Google will end up being best for non coding tasks. Google can concede coding to other vendors who pay Google for infra.
They can release even Jambon and Jargon, why should I care when I cannot test it?
Creating the hype
Don't bet against Google
People always love to think the AI race is a duopoly (OpenAI vs. Anthropic), but from what I have experienced, Gemini was always holding on its own, and I find myself using it more than Claude.
True, I use LLMs as glorified search engines to unearth material to read myself, but Gemini is much more useful if you decide to learn yourself. It also offers interesting expansions to your questions.
While I don't offload my work to LLMs (e.g.: Write this project, I'll be watching TV), I find them useful for closing my knowledge gap by collating information, and Google's Gemini is good at that.
Gemini is just too expensive for what you describe. Their 20$ plan only gives you access to the free tier model
They have an insane amount of resources but are so bloated, have a ton of bureaucracy, and have lost a lot of their top talent. Looks like they have figured things out now though.
Thanks David Guetta!
A lot of people do though by looking at the current stock price.
Never bet against their contempt for paying customers.
Really? Been 3 years since ChatGPT and Google still going from misery to another in their AI products.
Even if these numbers are correct, they don’t have any useful product for coding or search. Not sure what you’re referring to
Not everything is writing code and search. My uncle works in mortgages and the (very large) firm he works for is entirely Gemini. The way he talks about it is with wonderment in his eyes while he goes into detail on how it went through multiple documents and found unaccounted for cases. I think Google is doing fine in these kinds of industries.
I wouldn't even bet that Google is better than OpenAI and Anthropic for these categories, but they seem to be well used outside of our direct space.
How can one access Carbon? I only see Argon via gemini.google.com
Is this API-only or US-only for now?
https://blog.google/innovation-and-ai/models-and-research/ge...
"We're taking a deliberate approach to rollout and are actively engaged in the U.S. government’s voluntary process for pre-release model testing. Starting this week, we will make Carbon available to a subset of cyber defenders and will gradually phase access to additional trusted partners through our Fairwind Program. We’re continuing to gather feedback and iterate on guardrails before making Carbon available to developers, businesses, and consumers via our product surfaces as soon as possible."
It’s available only to those who read the announcement.
Even Haiku is better than Grok
I don’t care and in my opinion AA does a disservice to their users by allowing such blatant manipulation. What would happen if every AI company were to include their unreleased models like this? Put the unreleased models on a separate page and leave the headline benchmark rankings for models that you can actually use.
AA (like most other VC-funded benchmark sites) is pay-to-play.
Want a private benchmark and see where you'd rank? $$$.
Want to show up day 0? $$$$.
Want to show up day -0? $$$$$.
By the time this model is available for public use I'd imagine it won't be #1 anymore, which is pretty sad. Even when Google gets back on the ball they seem to find a way to fuck it up.
Nice coverup for the Q3 results. Wall St. don't give a F*
It's unobtanium so nobody should care about it. Any frontier AI company can nowadays create models with ridiculous resource and token usage that will get extremely high scores, but they're of no use when you can't use them.
I take this opportunity to give feedback on Argon. The limitations are a problem. I have a Pro subscription and Argon already stops working after 3-4 prompts with long delays afterwards. It's virtually useless right now even when you pay monthly for it. It is better for some tasks than 3.1 Pro, though. Not much in my experience but it can sometimes fix errors that 3.1 has problems with. I never use Flash models since they often give abysmal results.
So I'm stuck with using Gemini Pro 3.1 for now.
I couldn't care less, honestly
I have pro account and previous model "Argon" still not available on my account.
Mine showed up after clearing cache for some reason.
is it on web interface or antigravity ?
gemini.google.com (Pro plan).
AGY never even update often, and only has gemini 3.8 flash (for Google model).
A secret model I trained last afternoon is the actual #1 on benchmarks.
No, you can't use it. You can't even see it, and I won't provide you any information. But you must trust me that I am the genius who made the best model in the whole world.
well, $GOOG is down 13%...
slick
Spare me the celebration. Artificial Analysis should not be granting any company the favor of a #1 ranking for a model that isn't released to the public.
Google's model release is pure PR puffery. It's no coincidence that it was released on the day of its earnings call.
And i guess people dont care by google releases anymore
Argon did not leave a mark or change anything
Why would anyone care about carbon as well
Sunder quoted their tweet in his release announcement. Like that was the main thing he wanted to say.
Of course he would, he paid a lot for the privilege.