NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
▲Why is Gemini 4 Carbon taking so long? (kernel.dev)
tristanj 14 hours ago [-]
It's pretty obvious that Gemini 4 Carbon is not token efficient. It also has relatively low tokens-per-second compared to other models. I can think of several reasons why this is the case:

1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.

2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.

3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.

kyrra 6 hours ago [-]
1) "Gemini 4 Hydrogen is adapted from Gemini 4 Carbon." From https://deepmind.google/models/model-cards/gemini-4-hydrogen... . Also, the Agy team is pushing is that Carbon is a great planner, and then use hydrogen as the worker. They want people to use it.

2) lots of people inside Google use carbon. If they didn't care about it, it wouldn't have been pushed on all of Google.

3) Deepmind is competing with cloud and others. they are the only Frontier Lab that actually has to compete with other business units within an org for compute. GCP and others want that compute as well.

(Googler, opinions are my own)

zamadatix 5 hours ago [-]
1. Small, efficient models are often derived from "the best" models. That doesn't mean "the best models" are efficient too.

2. GP said they don't care about cost, not they don't care about the model

3. Competing for existing compute + being able to assign the projects own capex investment for new compute provides more compute than needing to assign all compute via new capex investment. But it's hard to say if that's actually the type of case or not unless Google wants to tell everyone publicly - so I'd take that point with a grain of salt in either way.

aberoham 11 hours ago [-]
Google's frontend web servers routinely time out connections from agy (the antigravity cli) when using Carbon, wasting tons of work. Like any effort level more than "low" is completely unusable.

It looks as if Carbon often takes more than 240 seconds thinking, which I'd guess appears as a silent connection to Google's load balancers, which then drop what it thinks is a stalled connection.

One hopes that Gemini's serving stack is smart enough to cancel these torn-down connections, but maybe they're not and this is leading to all sorts of ghost TPU load.

Disclaimer: We spend millions on GCP per month and have a dedicated customer engineer. Even they are horrified at the glacial pace of the Gemini fixes.

p_l 11 hours ago [-]
This has been acknowledged by recent "agy" release notes, it was timing out in client not in backend
elij 3 hours ago [-]
Yup I use carbon hours daily with 10+ minutes between requests at the extreme without timing out. That timeout has to be the harness. I also haven't seen API errors since release day.
yard2010 8 hours ago [-]
I never got any google model to work using their infra - 429, billing errors and timeouts. As someone wrote here a few months ago - it's an exercise in managing frustration. So I had to resort to openrouter.
kingleopold 9 hours ago [-]
this happened even before agy btw, their CLI was subpar level for months even before 3.1 pro release and now too
embedding-shape 8 hours ago [-]
Lots of Google tooling is the same for me, simply unusable whenever I try even simple stuff.

I think no one tests this as a "non-Google employee" or something, they all have specific testing backends or something, and once deployed to the public, there is no one responsible for the experience of non-Googlers and no one testing it, that's why most of the Google stuff has such poor UX because of hundreds of small cuts all over the experience.

antihipocrat 7 hours ago [-]
Google never had the mandate to align design standards across internal and external tooling. BigQuery and some of the other analytics tools are great though, I'd be surprised if staff didn't use the same internally - maybe someone here can confirm?
kccqzy 4 hours ago [-]
Staff used Dremel and then F1. At least when I was there. I only heard non-Googlers praise BigQuery.
kyrra 6 hours ago [-]
What specific things are like this? If you have a complaint and it's something I can reproduce, I can attempt to file a bug for it.
htrp 4 hours ago [-]
don't bother... you'll just end up getting frustrated when you realize no one in the org cares.
thanh81646 10 hours ago [-]
[flagged]
npn 7 hours ago [-]
4. they are so behind that they can not create effective datasets like other frontier labs do. So instead they do all the tricks the Chinese labs do: using RL with the single purpose of benchmaxxing the AA score.

Look how all Chinese models, or haiku/luna class US models behave, you can see the same pattern: they think too much and only work well with a subset of tasks

Doohickey-d 7 hours ago [-]
Which is weird, because Google is in the best position for getting training data: they have YouTube (which I guess is why Google's models are good at multimedia), as well as the Google crawler. Which effectively websites cannot opt out of if they want to be Google searchable. So Google's bot probably is the _only_ bot (scraper) that hasn't been blocked from a large portion of the internet.
htrp 5 hours ago [-]
Google legal refuses to allow them to do so.
icepush 4 hours ago [-]
Seriously ?
efromvt 6 hours ago [-]
I think raw base text training data may be decreasingly important vs good RL environments/gyms, and maybe less clear that Google wins by default there?
giancarlostoro 7 hours ago [-]
Not only that, they pioneered some of the key technologies... Who isn't still on transformer models? Also funny when you consider Meta is in a weird similar boat, introduced key things, so far behind the frontier.
aswegs8 6 hours ago [-]
The innovator's dilemma. Large companies are structurally worse at innovation.
adventured 5 hours ago [-]
To go along with that they're extremely slow (massive orgs) and don't feel their existence is at stake.

If GPT slips meaningfully behind, OpenAI is dead. The same goes for Anthropic. Both would die extremely quickly.

If you're within Google that is not the sense you have about the current context, nor within Meta or ByteDance or Amazon or Microsoft. They certainly view all of this as very important, they don't think they'll die within a year or two if they don't stay on top of the frontier. Meta has run from behind endlessly, as has Google, and their businesses are doing just fine (for now). If OpenAI were in that position, there would be no reason for them to exist and they would be dead by now.

Microsoft likely doubts their existence is at risk just because they can't field a frontier model. For now they don't see Office / Windows being killed off as a short-term risk (plus they have Azure and other products).

AWS is extraordinarily difficult to replicate, it's more akin to trying to compete with dozens of steel mills (industrial infrastructure) than a mere piece of software. And the rest of Amazon's cash spigot is advertising on its retail platform, which is facing close to zero threat from the frontier race, so Amazon can take its time and endlessly screw-up on AI (and so it has).

A frontier LLM can't manufacture a smartphone or laptop for you, and certainly can't manufacture 100 million of them with a zillion dollar manufacturing contract with TSMC et al., Apple isn't particularly worried. They want intelligence on their platform, they're not overly concerned about needing to own a frontier model. They can afford to be patient and get it right.

Capital is more powerful than AI. The capital owns the GPUs the AI requires, the capital owns the datacenters, the capital owns the grid, the capital owns the generation, the capital owns the factories, the capital owns the machines that produce semiconductors, the capital owns the patents. Apple, Microsoft, Google, Amazon, Nvidia, Meta, etc control the capital, financially they're far more powerful than most mega banks these days. And if Apple needed $100 billion for a project, it's one human phone call to Berkshire Hathaway. For now AI is feeble standing next to capital.

AI is the little bitch of capital, to put it bluntly. AI is on its knees begging capital for more compute: please sir, may I expand?

pfannkuchen 5 hours ago [-]
Well “they” didn’t, people who work for them did. Are any of those people still at Google?
owkman 6 hours ago [-]
Seeing how every new benchmark that has come out has Carbon doing well, I'd have expected the benchmaxxing allegations to die out. But they've been ok incredibly persistent. It's jagged like all frontier models. You could argue about it's warts, but benchmaxxing is unfounded. Did it benchmaxx every public and private dataset? Is that what RL is perhaps?
jckrichabdkejdb 5 hours ago [-]
[dead]
unbrice 2 hours ago [-]
[Citation Needed]
fsh 13 hours ago [-]
4. Google did a worse job than Anthropic and OpenAI.
dpe82 13 hours ago [-]
Well sure, but parent is trying to reason about why.
andy_ppp 12 hours ago [-]
It’s unlikely to be deliberate tbh, it’s more likely the data, algorithms and the training environments aren’t quite as good as Anthropic, Open AI is a lot closer to Google than people want to admit.
exe34 9 hours ago [-]
Why would data and training environment lead to frontier intelligence but slow?
andy_ppp 9 hours ago [-]
Because the environment did not reward being as efficient as other models?
exe34 9 hours ago [-]
Oh so it's using more tokens to do the same work. Make sense, thanks!
LoganDark 9 hours ago [-]
Why would a search engine that's had nearly the entire internet for decades not have the data of a lab that's only been scraping for a few years?
PaulHoule 6 hours ago [-]
The web is mostly trash. Go look at commoncrawl!
andy_ppp 9 hours ago [-]
Because Google are probably more hamstrung by bureaucracy and little things like the law.
LoganDark 8 hours ago [-]
Since when? Google recently disobeyed some legal requirements because "haha, couldn't catch me / too late now bitch". (This is when they cut down a forest for a datacenter)
tibbar 7 hours ago [-]
Internet data is only good for some parts of model training. That is why there is now a huge cottage industry that creates custom datasets, RL envs, etc.

Incidentally, Google is often considered to be extremely good at pretraining, which is just the thing internet data is good for.

andy_ppp 6 hours ago [-]
Yes I'm very much talking about data used in the RL section, I'm sure Google are indeed very good at pre-training.
htrp 4 hours ago [-]
Too bad pretraining hasn't been relevant for model performance since 2025
zarzavat 13 hours ago [-]
It's 1. Carbon was announced the day of Google's earnings call. It's a PR stunt.
kyrra 6 hours ago [-]
Googlers had been testing carbon for some time prior to the announcement.
Danywalery 7 hours ago [-]
Google has a huge flow of user data; they were the first to figure out about the Transformer, yet they’re lagging behind in the race. This is very strange, although their flash models are quite good, they’re expensive compared to the Chinese ones.
vintermann 11 hours ago [-]
So in short, as is usually claimed about Google products from bazel to tensorflow to Kubernetes: they may work better inside Google than outside of it.
serioussecurity 5 hours ago [-]
(2) isn't right. Cloud is on track to become the largest source of revenue and a lot of that comes from selling inference.
moffkalast 7 hours ago [-]
Maybe they should be more efficient and sell fewer ads.
teknologist 13 hours ago [-]
Also, Google uses their own TPU architecture rather than GPU/Nvidia.
horacemorace 14 hours ago [-]
I was just using this one today for some coding tinkering and it was super slow. But the way it solves problems with a light touch while still driving toward sensible architecture is refreshing after claude’s bulldozer attitude. It was able to integrate repeated creeping scope nicely with what it had already done. Very different feel than sonnet/opus, and even gemini 3.8.
artdigital 8 hours ago [-]
My thinking as well! I really like Carbon

It’s hella slow for sure and agy harness makes this worse by telling it to read a ton of files before making any changes!! But once it starts, its solutions are very good and it usually continues until the task is fully completed! The output was so good at times, I was honestly surprised by how thought-through and properly implemented everything was. It implemented edge cases I would have missed and made decisions that I would call sensible. For example adding a —-flag that absolutely improved the product.

Code review still catches medium priority issues but that’s the same with all of the models.

I’m now putting Carbon on bigger tasks that I don’t care how long they take. As I am writing this I’m having dinner while Carbon slowly chops away at a task queue on my server.

All in all, Carbon is great, Hydrogen is great, Google is killing it!

(agy is still bad though, but you can tweak the “read a million files first” behavior with your system prompt)

joshuanapoli 7 hours ago [-]
My experience is similar: Carbon found better ways to solve open-ended problems than Fable. In these cases, where I don’t have a solid design ahead of time, it’s OK to wait.
dhruvmittal 1 hours ago [-]
Definitely agree with this, I regularly find that carbon writes less overengineered code and considers more of my existing architecture without additional documentation or prompting. Maybe that's what it's doing with all of its "wasted" turns?

That said, it's incredibly slow. I haven't played much with the orchestrator that agy added but if it's Carbon occasionally delegating to Hydrogen, that might be a little backwards. I feel like what I actually want is Hydrogen delegating more difficult tasks to Carbon instead.

weatherlite 11 hours ago [-]
So it's slow but good? Reasonable trade off I think
antupis 9 hours ago [-]
it depends but IMO Carbon is best at the moment with code reviewing and documentation writing. Opus is still my go-to model for code generation but at moment Carbon does better job for those auxiliary jobs.
eknkc 9 hours ago [-]
Why not Fable? I've mostly been using GPT side models and have recently been dipping my toes into Claude models. I went big with Fable 5.2 as it was "the best" the offered. Apart from token usage, is there any particular reason to use Opus?
antupis 8 hours ago [-]
Our organization still blocks Fable because reasons probably something related to storing prompts.
glimshe 11 hours ago [-]
It depends on how much you're willing to wait. I used Argon a few times and while it did some interesting things, it was so slow that I couldn't fit it in my workflow. GPT Sol was great and 5-10x faster.

I feel that right now Hydrogen is the only viable model Google has in cost/speed-benefit. It's cheap and makes sense for general use in the $20 month plan with Youtube and storage, but not as a prime coding model.

bagacrap 4 hours ago [-]
In case folks missed it, Google essentially recommends using carbon and hydrogen together: https://antigravity.google/blog/gemini-4-hydrogen-in-google-...
thisisauserid 4 hours ago [-]
Using them together is faster and cheaper and not quite as good as just Carbon (if I read that right). So that's a pretty good sweet spot for most use cases.

Gemini gives you a lot of options.

eknkc 10 hours ago [-]
[dead]
nautilus12 6 hours ago [-]
I agree, the best way i can describe Carbon is that it is "cautious". Very very refreshing attitude for a model to have.
hetspookjee 7 hours ago [-]
Might be the only one in the thread but ever since Gemini 4 Argon. I’ve become a huge fan of the Google models. I do use Carbon most of the time since it’s become available but notice that i also am falling back more and more on Argon again without much difference. For security purposes argon has far fewer triggers. I only use the Google models as orchestrator for how they talk and analyse. I let it delegate everything to Astra / Fable / Opus models
eitally 1 hours ago [-]
I've also become a fan. For most of my current projects I'm using Agy with the Carbon orchestrator, but switch to Hydrogen when I'm bantering back & forth. It's convenient, with things like Swarm, to have an orchestrator that can automate the selection of models for sub-agents. I've used quite a few harnesses -- both for local and cloud dev -- and although Claude Code has some clear feature benefits over Antigravity (for desktop/sandbox control & interaction), I'm generally very happy.
wronglebowski 3 hours ago [-]
I do like Argon, but usage continues to be an issue for me. The economics of everything pales in comparison to the value you get with DeepSeek. I'll end up using my limited opaque usage based limit with Antigravity to have Argon review something or maybe write a plan, but the bulk of the work has to be done with cheaper models.
hetspookjee 3 hours ago [-]
I use it similarly but have the subscriptions to do other worker models. But yea I tell it only to delegate except when the change is something tiny. But most of it is done n via delegation.
soared 3 hours ago [-]
Similarly $20 on opus in Claude code seemed to last me 10x longer than argon in antigravity. And for some reason there doesn’t seem to be an opus equivalent in antigravity - but in Gemini I can select “pro” which doesn’t exist in antigravity (flash or argon only). Antigravity $20 plan also gives a comical amount of Anthropic credits… it can literally only answer one question for me before hitting limits.
chubot 5 hours ago [-]
What harnesses do you use, to switch back and forth between models, and to orchestrate?
hetspookjee 3 hours ago [-]
I tell t to send it to Claude -p opus 5.5 and or Astra. I made the model mappings explicit in GEMINI.md. I don’t ever leave agy anymore and anytime I need to I dislike how Astra and Claude always beat around the bush
WarmWash 5 hours ago [-]
I haven't used Carbon or Argon outside antigravity, but historically Gemini models take a bad hit outside of it.
htrp 3 hours ago [-]
So basically the Argon is your scheduler and then the other frontier models actually accomplish the work?
xnx 7 hours ago [-]
There's dozens of us!
moffkalast 7 hours ago [-]
That's right, you are probably the only one :)
fwee 2 hours ago [-]
Carbon is slow, but I've also realized that it works really well. I don't need to get involved or keep checking the results over and over again. I think it's pretty close to Opus 5.5 in terms of work quality.

I have no doubt that Google will adjust its training to find a better balance between speed and intelligence, just like they did with Gemini Flash (3.7 -> 3.8, slower but smarter).

The orchestrator and swarm modes are also interesting. Even though Carbon is slow, it can delegate tasks to other models. (They're both as slow as Carbon, but I think they have potential.)

Anyway, great article!

bagacrap 4 hours ago [-]
This part actually seems like a feature:

> Unnecessary work. About half of Carbon's steps went on work the tickets never asked for: auditing code the fix didn't need and closing loopholes the ticket had already ruled out.

I have anecdotally noticed this, but I thought of it as Carbon "thinking" more "deeply" about the implications of a change. That can be a good thing! Instead of just doing exactly what I ask it, it is able to flag new bugs in far away code that would result, and it's able to find very subtly dead code in large codebases during redactors. My daily driver is Sol, which is exceedingly fast but is relatively focused on the task at hand. I think of my collection of models as similar to a team of humans who all have their strengths and weaknesses, and I'm much more likely to ask Carbon to review some work I've done than to help enact my ideas in the first place.

jedbrooke 3 hours ago [-]
I noticed a similar issue going from Opus 4.6 to 5.5. I also however upgraded my harness during that time, and I found out that it was injecting something like “Be helpful, read in between the lines and proactively go beyond the user’s request” at each turn. I immediately patched that out and saw decent improvement in that area. TFA mentions they tried different harnesses though, so yeah it’s prob baked in to whatever RL setup they’re using to train the original model.

This is why I feel like instruction following is really all I care about in a LLM. I just want it to do what I asked (not what it thinks I actually meant). Intelligence can always be supplemented with tool calls, but an LLM that won’t do as it’s told is no good

ghm2199 3 hours ago [-]
This. Yeah I think if one knows what they are doing, e.g. a seasoned developer. They can get a lot of mileage out of an LLM that is good at instruction following. which is why i like codex models. Astra may not be as good as Opus but the workhorse really is 6.2 sol. The best workhorse when it comes to the pareto frontier of `cost vs accuracy` or `inference time vs accuracy`.
totalhack 2 hours ago [-]
Carbon has been slow, but excellent at deep dive analysis of some of my projects, finding optimizations other models missed AND explaining them in a clear and concise manner, even as the conversation develops and really becomes four or five subthreads on the opportunities that it brought up. Maybe that massive context window helps.
bugzbenny 2 hours ago [-]
Using any of the latest models within Antigravity is incredibly slow. I use Claude Desktop for sending off coding agents into the cloud and I get about 10x more done with it than I can do with Antigravity + Google models. The only advantage to the Google models over Claude so far that I've noticed is they are way better at writing copy.

Anecdotally, the Google harnesses (Gemini web/desktop, or Antigravity) tend to lose connections way more than their competitors.

I wanted to like Google's offering (since all my data is there) but I feel as though they've really dropped the ball.

Otterly99 2 hours ago [-]
> Closing loopholes the ticket had already ruled out. Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.

This is unfortunate. This was one of the most annoying behavior that appeared at Opus 4.6? 4.8? Not sure exactly, but one of the post 4.5 version started to be very proactive, on top of being very verbose. Thankfully, Anthropic did listen to their users and improved things greatly on this front.

It seems that Carbon is designed to have the same flaws, I would advise Google to listen to their users and get rid of the verbosity and proactivity.

timtam 10 hours ago [-]
Using it through the prompt box, I kept running into 500 INTERNAL (TURN_STEPS_EXCEEDED). Gemini couldn’t finish this response I’m sorry, this request needed too many steps to finish in time. Please try asking a more specific question, or break it into smaller requests.

I only got around the issue by strictly setting an effort budget or a stopping rule.

seizethecheese 3 hours ago [-]
We have only 146 battles [1] with Carbon, so I kicked off an analysis with skepticism we'd have anything to add here. I was wrong. Carbon is that slow.

Biggest differences:

- Time to complete: 6m 08s for Carbon vs 2m 04s for opponents.

- Output tokens 16.4k for Carbon vs 6.8k for opponents.

- Tool calls: 21 for Carbon vs 11 for opponents.

All three of these are statistically significant.

The ones that don't seem to be a huge issue, but have been mentioned here or in the blog are inference speed (52 tps vs 73 tps) latency (12s vs 8s) and agent turns (13 vs 8).

Interestingly enough, Carbon has a high win rate when the voter waits for both solutions to finish, but a terrible win rate from early voting.

- Both finished 83 carbon wins, 52 losses.

- Early votes 2 carbon wins, 40 losses!

Basically, Carbon is good when you're willing to wait.

[1] we're building a battle system on top of opencode. leaderboard: http://agentduel.com/leaderboard app is open source: https://github.com/bottomless/agent-duel

2 hours ago [-]
lmeyerov 6 hours ago [-]
It's been jarring & confusing to go from a year and a half of Claude Code max plan, and equivalent of Codex, to Google agy & their models

My flow is roughly a few days to max out Claude sub, a few days to max out codex, and 1 afternoon to max out agy

While Astra/fable, sol/opus, sonnet/Luna are largely interchangeable when jumping codex vs Claude Code on the same repo+skills... that hasn't been true when using agy with Google models

My next stop may be drop agy harness in favor of codex/opencode to see if the models are fine and agy is the problem

Lerc 13 hours ago [-]
A Theo vid suggests there's a time to first token issue. When doing subagent actions it can compound to be massive from a bunch of internal short queries.

It really looks like something is broken more than just being inefficient.

xnx 7 hours ago [-]
How recent is that video? Google just reduced time to first token (https://x.com/_mohansolo/status/2122821676249625014): "Time to first token (TTFT) for Carbon is now 1.5x faster on Antigravity. Will follow soon with Argon too.

Lots of work went into this so you all could have a faster experience. More to come "

96khz 13 hours ago [-]
are we talking about token prefill here?
Lerc 12 hours ago [-]
11 seconds worth?
teknologist 13 hours ago [-]
Theo seems to dislike anything that isn't Anthropic Claude. Suspicious, to say the least. On CursorBench, a good measure of real-world coding tasks, Gemini 4 Carbon comes out top: https://cursor.com/evals
LiamPowell 13 hours ago [-]
Every time I see a Theo tweet he's judging something on the most absurd metrics. Today for example, he was saying Astra 6.2 is bad because when he gave it a single line prompt telling it to create a new level for a game it used a name for the level that he'd seen before. Judging a model on how often it reuses a name is one of the most ridiculous benchmarks I've ever seen.

As far as I can tell, all his judgements are based on similarly absurd metrics, I don't know why people treat his preferences as valuable.

4 hours ago [-]
ywvcbk 7 hours ago [-]
They have excluded all OpenAI models, though? Of course I doubt they'd have the highest scores but likely would be most price efficient.
teknologist 4 hours ago [-]
No, they have not. But they don't seem to have the latest ones. I think that's because OpenAI announced that they will stop providing their models to Cursor.

https://openai.com/index/our-decision-on-cursor-following-it...

spprashant 5 hours ago [-]
I do not like Theo, but he is transparent about his OpenAI bias. He criticizes Anthropic frequently - mostly for their hostile closed-garden policies.
knollimar 8 hours ago [-]
I’m not sure Claude favoritism is fair; he loved OpenAI for a while. He just has really arbitrary and frankly bad takes imo
niltecedu 3 hours ago [-]
I am pretty sure caching is busted, atleast on github copilot, I tried doing it via pi harness however cannot seem to get any cached inputs post 5 minutes.
7 hours ago [-]
SubiculumCode 14 hours ago [-]
That isn't a good look, and either this is unmeasured uncontrolled behavior, or intentional under some longer term plan...but the latter possibility seems improbable.
ahsillyme 14 hours ago [-]
Uhm, I'm not a programmer but why isn't this a good look? I would want my engineers to fix problems that they notice in passing. (Edit: the "unnecessary work" angle is what irked me in the article)
horacemorace 13 hours ago [-]
Good SWEs get a feel for the minimum amount of changes to fix something. Fewer changes usually minimizes the chance for new problems to pop up. Junior SWEs usually try to do too much.
bot403 11 hours ago [-]
This. There should be 2 PRs. The core fix, the the unrelated fixes. Because I guarantee sometimes those unrelated fixes will cause regressions so they have to be scrutinized too.
shwetanshu21 11 hours ago [-]
Sometimes that problem might have downstream other effects which causes bigger problems which is why it is left as it is.

Most of the times it has to do with refactoring downstream services or dependencies or just unexpected behaviour that shows up.

lp92 14 hours ago [-]
>Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.

That's pretty interesting. Seems like it's over analyzing. Not the worst outcome though to have additional hardening.

With that said, I found Argon to be better for most coding tasks and Carbon for architecture/deep RCA analysis of complex issues.

gazebo2 3 hours ago [-]
I find this behavior to be common across most of the big models -- Sol 6.1 does it a ton, adding lots of super defensive programming cruft for things that A) by design can never happen and B) should fail loud anyway, despite prompting. It gets a little better after steering tweaks but still common.
dada216 6 hours ago [-]
ITT it's that huge max output token of 1mln tokens.

It's pretty far beyond other models (Opus 5.5 is at 128k).

0xbadcafebee 1 hours ago [-]
Gemini is consistently the worst performing AI product, agent, model, etc I have used. I don't really understand who keeps using it.
jaren 2 hours ago [-]
any hypothesis why slower inference?
mengqijin 11 hours ago [-]
I tested with long and complicated tasks Carbon did perform better and took almost the same time as Hydrogen, roughly 50% more cost than Hydrogen.

But with simple tasks, Carbon is way too slow, and much higher cost.

5555watch 7 hours ago [-]
Carbon is extremely good for particular research questions I'm doing, and the only hope at the moment is that they don't eventually nerf it down, especially if topics like these about the speed might raise concerns.

In my several anecdotal cases it's miles ahead of Astra Ultra & Opus 5.5/Fable 5.1 (haven't tested on Astra 6.2 and Fable 5.2 yet). Importantly, the Astra and Opus criticizes the ideas proposed by Carbon as bad, they both use the same critiques. While from my eyes the Carbon makes good points.

So recently I've started ignoring Astra & Claude, and said, go on Carbon, finalize the proof of concept and let's see your results. Few nights of calculations, they do some good work in my eyes, I show the results and workflow to Astra & Claude, and only after seeing it all they say "This is substantial progress!". So yeah, duh!

Two points: First, it seems Claude and Astra are super correlated models; Gemini is less correlated.

Second, they criticize everything so much, that these critiques are evident in their workflow - they're too careful to do something risky, so they can't find good solutions if they're not super obvious. And I think if referees will be using these models to write reviews, they will nitpick on tiny details and will always find dozens of them.

joshuanapoli 7 hours ago [-]
Yeah, Carbon is good when the problem needs some imagination, when a better solution is worth waiting for. Fable 5.2 moved towards faster or more efficient responses. So it’s more practical for most tasks, but tends to be less creative.
oytis 7 hours ago [-]
They claimed a precious domain I should say, shame that they are not using it for anything kernel-related
timcobb 7 hours ago [-]
really glad to see _this_ kind of analysis of these models. These type of analysis is much more practically useful than some context-less (especially with regard to time) number of "how good it is".
ahmadyan 11 hours ago [-]
i'm having all sort of problems with the gemini carbon lag. using it through open-router is impossible (at least for me), the request times out after 5mins, and often it take more than 5mins for the first token to arrive. I don't know what batch size they are using, but this is not useful.
cybice 11 hours ago [-]
Interesting that it looks like behaviour of cheap models, like deep seek flash overuse tool calls
jrflo 14 hours ago [-]
Is this due to Google wanting to catch up on benchmarks at all costs, and trains their models to be overly meticulous to do so?
murderfs 12 hours ago [-]
I'd guess that it's because Google eats its own dogfood, and if they didn't train the models to be overly meticulous, the existing codebase would implode under a tsunami of slop.
WeZZard 5 hours ago [-]
I didn't try but some people said Gemini 4 Carbon served by VertexAI is faster.
4 hours ago [-]
user_7832 14 hours ago [-]
I am increasingly more convinced every day that every LLM company is just making their latest model to be the best at (coding) benchmarks, at the cost of pretty much everything else.

I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.

Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).

vintermann 11 hours ago [-]
I also use LLMs for a difficult non-coding task (historical HTR), and my experience is that Gemini Argon is much better than its competitors on that.

Its main failure modes are silently stopping with the first column of a multi column layout, and ignoring explicit formatting instructions and polluting the transcription with letter variants and ligatures (such as long s, ū, double-dotted y and æ in Latin). It's a bad habit it's picked up from HTR datasets.

Still, those are much better failure modes than hallucinating, which powerful models are still prone to.

incognition 14 hours ago [-]
Oddly flash 3.8 for me is good. they really pushed carbon to some other extreme
p_l 11 hours ago [-]
Heard a rumour that Carbon is ~10x the size of Opus 5.5
netrap 4 hours ago [-]
I liked flash as well but it is gone now?
p_l 3 hours ago [-]
Still have 3.8 Flash available here
SyneRyder 12 hours ago [-]
Are you able to give hints to some of those use cases? Recent Opus and Fable seem to have made big leaps in art and music, and I don't think all of that is the longer context window since Sonnet 3.x. But those are both things that can be created with code and mathematical thinking as well. I definitely wouldn't want to use recent models for writing though (not that I ever wanted them to write for me anyway).
knollimar 8 hours ago [-]
Opus 5.5 trashes everything not fable and astra for my knowledge work. It’s an insufferable pedant though.

I haven’t used a lot of Carbon though.

auyez 13 hours ago [-]
At this point it might make sense to put all their resources into improving coding and math skills, because it might lead them to a state where they keep accelerating their capabilities. Even in the worst scenario where LLM approach would hit some wall, they would have technology to quickly explore all other approaches.
londons_explore 12 hours ago [-]
The Human mind so far has shown remarkably little ability to redesign other human minds to be smarter.

The existence of AI able to design a smarter AI I would not say is a given.

wellcolormesurp 13 hours ago [-]
They are cracking software and math, not finance, law or medicine or whatever it is you do.

Those two domains have leverage, the others do not. It makes perfect sense. If you and I had the goals they have we would start with code and math as well. The other dominos will fall automatically once you got the engineering automated. It makes no sense to stop focusing on that until you either run out of money or achieve your goal.

oblio 9 hours ago [-]
> The other dominos will fall automatically once you got the engineering automated.

Based on...? Wishful thinking?

resters 8 hours ago [-]
i wanted to test carbon but ran out of quota before i could get a definitive result.
gdevenyi 6 hours ago [-]
I just got a 6 month max subscription via the antigravity science program. I've tried to start using it and found both the harness to be terrible and the model to be very slow.

It might do good work but I'm having a hard time figuring out if it is.

nandanadileep29 12 hours ago [-]
yeah i tried using it its not efficient at all. just burns up the token and not get the work done.
logicchains 11 hours ago [-]
Still gives far more value for $200 than an OpenAI subscription; Astra burns through limits way faster than Carbon, and Sol makes way more mistakes than Carbon (and Opus 5.5). Maybe it's only temporary though; the Google $200 subscription gives 4x then $100 one, while the OpenAI one only gives 2x.
kristjansson 12 hours ago [-]
Argon isn't much better. It just. keeps. going. OTOH when I asked it (in the agy cli) about some detail of how the permission system it was neck deep in decompiled go by the time I lost patience and killed the turn. So at least sometimes it's doing something with the time and persistence?
p_l 11 hours ago [-]
[dead]
12 hours ago [-]
dario_amodei 13 hours ago [-]
how much?
coldbreeze485 7 hours ago [-]
[dead]
ath3nd 52 minutes ago [-]
[dead]
sodowork 13 hours ago [-]
[flagged]
georgestones 13 hours ago [-]
[dead]
erji77 4 hours ago [-]
You idiot, stop pushing AI news to me from now on.
NoahZuniga 13 hours ago [-]
Google pretty clearly says in their product documentation that carbon is just a version of carbon that thinks way longer.
lukax 12 hours ago [-]
You probably meant a version of Argon?
kzrdude 12 hours ago [-]
Then that's "test-time scaling" again and Google then showed that they can reach frontier level benchmark results using it. It's pretty cool as a technology actually (the demonstrated capability), even if the model is not so useful when it's slow.
joedwin 12 hours ago [-]
Argon right?
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 19:26:03 GMT+0000 (Coordinated Universal Time) with Vercel.