For people interested in this: I've been working on an open-source libtpu implementation for https://tenstorrent.com/ accelerators as a side project, it can already run most of torchtpu, see https://github.com/pcmoritz/libtt (based on the tenstorrent software stack). It shows all the things involved in lowering torch to tile based accelerators.
Thanks for sharing, this is super cool! Right now my approach to correctness it to just run as many tests as possible and fix them (torchtpu greatly expands the amount of available tests), but formal verification is also super cool. Having a look at your book https://lean.brettkoonce.com/blueprint.pdf
Thanks for the encouragement, if you have the hardware and end up trying it out I'd love to get your feedback (feel free to drop an issue in the repo) :)
yhjc2692 1 days ago [-]
I would love to do that, but unfortunately don’t have one on hand (yet?). In fact, I’m curious - did you buy a TT device to work on this, or are you using a cloud provider like Koyeb?
pcmoritz 1 days ago [-]
I first built my own to try it out (the exact hardware is described in https://relativity.cc/hardware/ under "one Blackhole card") and when I liked the hardware I got a quietbox 2. It is very high quality but a little expensive (but inexpensive compared to other LLM hardware and it has excellent interconnects which is very rare in the consumer/dev space).
dalemhurley 1 days ago [-]
Google and Apple are going to move consumers and ordinary business users to edge inference. This is more disruptive to the upcoming IPOs than most investors understand.
lofties 21 hours ago [-]
Google, Apple, and don't forget our friends DeepSeek!
anuptalwalkar 23 hours ago [-]
I would love to see them move off into edge computing. There should be some autonomy and siloed experience in day to day usage.
tjwebbnorfolk 23 hours ago [-]
I would love to be able to buy a TPU for inference
They are just wildly outdated for most tasks. Even projects like frigate suggest buying different hardware. Coral was released in 2019 and hasn't been updated once. With a jetson nano or even a raspi ai hat you get way more tops
tjwebbnorfolk 7 hours ago [-]
Ok I'll clarify: I want to buy a current-ish generation of TPU that can run LLMs. Coral is like 8 years old now
emmelaich 21 hours ago [-]
You can buy a Google Coral TPU, does that count?
Dr4kn 16 hours ago [-]
With them being released in 2019 with zero hardware revisions they are just outdated for almost every workload
villgax 9 hours ago [-]
Yeah right, try generating an image on Apple Silicon or an Android phone with any of the Qwen or Flux family of models, let alone videos or prefill heavy LLMs.
npalli 1 days ago [-]
Interesting, Amazon doing the same for AWS Trainium.
Pytorch conference is happening today for explaining the coincidental timing
sroussey 1 days ago [-]
That explains the 5x speed increases huggingface mentioned for Mac using metal.
mathisfun123 1 days ago [-]
Link?
fancyfredbot 1 days ago [-]
Cool! XLA is pretty good + potentially a better compiler infrastructure than inductor which will lead to improved performance on all hardware.
Also I guess this is the final nail in the coffin of Tensorflow.
23agt 1 days ago [-]
Google is selling TPU shovels via Cloud services and also physically to select partners. So this makes sense.
There will be an abundance of shovels. Nebius already crashed by almost half from its hype peak.
airza 1 days ago [-]
it doesn't seem like there's an abundance of shovels coming anytime soon
tjwebbnorfolk 23 hours ago [-]
Everyone says there's a massive compute shortage. But where is it showing up? For whom is there a shortage?
Every time I ask Claude to do something, it does it just fine. I never have to wait in any sort of queue that would be common in other kinds of shortages.
eitally 23 hours ago [-]
It's showing up for enterprises (and ISVs) who need capacity from cloud vendors and can't get it. Sometimes there are some GPU shapes available, but not always what you either need or want to pay for (e.g. a TPU vs a V100 vs a B300), and also not always in spot instances (frequently not).
The hyperscalers are all so oversubscribed that they require commits from enterprises so if you only have a transient need you might be out of luck [from your vendor of choice]. Additionally, not all GPU service providers are equal. Semi Analysis has some great articles explaining why.
octoberfranklin 21 hours ago [-]
But where is it showing up?
Pricing -- exactly where shortages show up in a free market economy.
tjwebbnorfolk 7 hours ago [-]
Right, but pricing is for the marginal unit of hardware.
what I'm wondering: is there a shortage of compute in the current installed base? Why isn't the installed base of compute increasing drastically in price, and/or forcing me to wait in a queue to use it?
airza 8 hours ago [-]
have you tried to purchase a device with RAM or a GPU in the last two years?
tjwebbnorfolk 7 hours ago [-]
Yes it's all very expensive. But is there not enough compute in general? Or is there not enough extra compute for me personally where I want it ?
like, I can rent a VM from AWS for the same price as 5 years ago...
bornfreddy 7 hours ago [-]
I mean there is an abundance of shovels - but everyone wants to dig with excavators, and there's a shortage of those.
juleiie 1 days ago [-]
Shovels and gold aren’t the problem, humans are the problem
either AI achieves hyper-efficiency and triggers systematic economic demand destruction due to unemployment, or it underperforms and sparks a massive tech-sector asset bubble burst
If AI is as good as valuations - it crashes. If AI isn’t as good, it crashes. all eggs in one basket economy.
Every future leads to a massive financial shitstorm
AI is priced in as a technology to end all markets. The last bull market you will ever see.
It’s a miracle markets didn’t notice this “damned if you succeed damned if you don’t” type of inevitable checkmate
I have been building a bunker
bsder 22 hours ago [-]
> It’s a miracle markets didn’t notice this “damned if you succeed damned if you don’t” type of inevitable checkmate
They learned from 2008 that Too Big Too Fail(tm) means the government bails you out.
This was the problem with not OBLITERATING the banks in 2008. Had the US government taken over the banks and wiped the investors, there would be more pushback right now.
omgwtfbyobbq 1 days ago [-]
It's possible. If goog's AI overview is accurate, it seems somewhat similar to speculative euphoria seen in the past (eg companies involved in electricity prior to the great depression). It's larger (currently 35% for AI versus 25% for electric/related cos), I'm guessing because of more credit. At the same time, the housing component of the market prior to the GFC was about 23%, and it wasn't as economically damaging as the GD, so hopefully the AI bubble unwinding won't be as bad.
esseph 1 days ago [-]
> so hopefully the AI bubble unwinding won't be as bad
It's currently larger than the 1880s railroad boom.
"Overbuilding and speculative debt ultimately culminated in the Panic of 1893, plunging the nation into a severe economic depression."
omgwtfbyobbq 23 hours ago [-]
Not according to Goog's. The railroad bubble of the 1880s was supposedly 60+% of the market. It may not be accurate though.
"During the 1880s, private railroad enterprises represented over 60% of the total capitalization of the entire U.S. stock market, peaking at 63% in 1881 and 1885."
esseph 6 hours ago [-]
AI spending is significant more in inflation adjusted dollars. (Roughly 550 Bn in modern dollars). We're projecting trillions in AI spend.
But as a % of total GDP, you are correct.
omgwtfbyobbq 52 minutes ago [-]
I asked Goog's for enterprise value of the sector as a percentage of total market in all 4 situations, so it's not exact because enterprise value isn't available back to the 1800s, but it should be ballpark assuming the overall picture is accurate.
In terms of infra spending as a percent of GDP, 1880s railroads were at 5-6% at peak while AI is estimated to reach 3+% in a year or two.
fasterik 1 days ago [-]
[dead]
ismailmaj 1 days ago [-]
Amazing news, I remember the days when you had to fight XLA to make PyTorch run on TPUs, it seems we finally got first-party support!
Since HW specifics live mostly in the kernel layer (Pallas or CUDA), I imagine the Jax vs PyTorch question is more of a style preference now?
nico 1 days ago [-]
Pardon my ignorance, but who else besides Google has TPUs? Are they commercially available somewhere? What’s their main application right now outside of Google?
cavisne 1 days ago [-]
You can rent them on cloud + they sell directly to Anthropic. Its pretty well known that Anthropic is a heavy user of TPU's (along with Trainium & GPU's)
atty 18 hours ago [-]
Google is selling TPUs to enterprise clients for on prem deployments. We’ve evaluated them but decided to stick with Nvidia, at least for one more generation. We’re a relatively large user of GCP.
Is a pretty successful open source NVR that used corals when it launched (and many people still use them with it)
qlte 15 hours ago [-]
Yeah I have one for Frigate on a Raspberry Pi. Has worked well but definitely showing its age performance wise. I got a cheap mini PC (right before prices like quadrupled) which has about the same performance on an iGPU). I wouldn't be surprised if Frigate was the funnel for like 90% of Coral purchases or more.
boulos 1 days ago [-]
Only in marketing name. They're not related otherwise.
goda90 1 days ago [-]
Maybe just marketing, but they call the NPUs in their Pixel phones "TPUs"
fulafel 17 hours ago [-]
TPUs are really matrix processors (they process 2d tensors right?), it's technically fair to call most matrix processors TPUs.
westurner 1 days ago [-]
TPU / neural compute accelerator / Math coprocessor;
> How does Cerebras WSE-3 with 44GB of 'L2' on-chip SRAM compare to Google's TPUs, Tesla's TPUs, NorthPole, Groq LPU, Tenstorrent's, and AMD's NPU designs?
There are TPUs made out of silicon carbide with standard fab.
There is in-flight tensor processing with photons, but that's not quite a TPU.
hackernudes 1 days ago [-]
I think the commenter was asking if Google sells TPU chip hardware to use outside of Google's own infrastructure and cloud.
g58892881 16 hours ago [-]
great! where can I buy a TPU?
tingkuanpei 9 hours ago [-]
Very interesting work. Google should have done this a long time ago. PyTorch's ecosystem is just so rich.
frobisher 16 hours ago [-]
Question: does this make JAX obsolete? I was having fun learning it!
csvance 10 hours ago [-]
It makes PyTorch work nicely on TPU, which is impressive mostly because of how unstructured PyTorch is with its conventions. From a compiler standpoint JAX is much easier to work with since its all pure functions. This means that JAX also has access to fairly cutting edge compiler tech like Enzyme which is quite relevant for scientific machine learning work.
nelaggy 1 days ago [-]
so if the moat isn't the model and the moat isn't the inference layer, i guess the moat is the hardware now? what does this mean for frontier labs?
badrequest 1 days ago [-]
hardware has always been the moat in the AI space
AnotherGoodName 1 days ago [-]
Yeah a lot of people bought into the idea of CUDA being a moat but we’re well past the point where it’s pretty trivial to jump to another equivalent stack on the software side.
jacquesm 1 days ago [-]
Those are the two main interfaces: the inference hardware and the endpoints. Endpoints are pretty much standardized to the point that you can switch them out in seconds. CUDA still has an edge but every release like this chips away at that. Even so, TPUs are unobtanium so while I appreciate Google's effort here I also recognize that it is not something that will be useful to me.
In that sense AMD is still better positioned to give NVidia a run for their money but given the family connection and AMD's underdog status I am not very hopeful that we'll see meaningful competition in inference hardware any time soon.
NVidia is a much harder nut to crack than OpenAI or Anthropic. They are easily a decade ahead of viable competition. Check out the GB300 if you don't believe that.
lp92 6 hours ago [-]
Google does provide TPUs to big players like Anthropic and other large enterprise customers for on prem deploymens. That's their target just like those the GB300 is aimed at.
kavalg 16 hours ago [-]
While I have the same sentiment, I am also curious why a competitor can't accelerate their developments with AI. In the end, hardware development is to a large extent also software development. Is it the patents that NVIDIA has that are preventing this? Or maybe some trade secrets? AFAIK the NVIDIA hardware is manufactured entirely by TSMC.
nixon_why69 18 hours ago [-]
A decade is a long time and the highest end isn't the only market segment.
High bandwidth flash will be here soon, I bet someone makes an economical competitor thats not quite as fast but much cheaper.
coredog64 1 days ago [-]
Even if it's allegedly possible, the devil has always been in the details. It always sucks to find out that there's some quirk of your model architecture that breaks an implicit assumption that underpins the export process.
yaoke259 18 hours ago [-]
[dead]
AndrewKemendo 1 days ago [-]
Will always be true
JacobAsmuth 1 days ago [-]
Products can also be a moat.
yipinwong 1 days ago [-]
Dumb question from non-AI engineer.
Is this somehow Google's answer to today's best DeepSeek open-sourcing inferencing engine?
bitpush 1 days ago [-]
I dont think big companies like Google react to news like that. Very likely this was meant to go out on Tuesday for weeks now. And it just happened to have coincided with another news.
JacobAsmuth 1 days ago [-]
Yeah they read the news this morning and whipped up a quick Torch->XLA translation layer and coordinated with four different inference engines (who also all wrote their own blog posts) and shoved this out the door quick to stay on top of the news cycle
parthdesai 1 days ago [-]
Why be rude on purpose? OP clearly prefaced by saying it was a dumb question
caaqil 1 days ago [-]
[flagged]
luckydata 1 days ago [-]
[flagged]
mathisfun123 1 days ago [-]
they're releasing this today because the pytorch conf is this week...
0x00cl 21 hours ago [-]
We might never know, but they probably didn't read the news and started working to get this out, they were probably working on it for a while now.
Personally the question is if they are "saving" other projects or news so they can be released in case a competitor makes a move? Who knows if this was ready days or weeks ago and were waiting on someone to drop some news, so they can shadow those news.
geremiiah 1 days ago [-]
No, these are two very different objectives.
Tanjreeve 1 days ago [-]
They do different things and sit in different parts of the stack (training vs inference). TorchTPU seems more like a play to get more people familiar with Googles TPU chips as they want to sell them to people outside of google.
beardedetim 1 days ago [-]
Yeah I'd assume this is to allow us to train on their TPUs with torch instead of needing to rent a GPU
yipinwong 1 days ago [-]
ty. that helps a lot and know where to research next.
The Tenstorrent ISA is documented btw https://github.com/tenstorrent/tt-isa-documentation, I wonder if one could do the verification end-to-end with the hardware.
https://github.com/aws-neuron/torch-neuronx
Also I guess this is the final nail in the coffin of Tensorflow.
There will be an abundance of shovels. Nebius already crashed by almost half from its hype peak.
Every time I ask Claude to do something, it does it just fine. I never have to wait in any sort of queue that would be common in other kinds of shortages.
The hyperscalers are all so oversubscribed that they require commits from enterprises so if you only have a transient need you might be out of luck [from your vendor of choice]. Additionally, not all GPU service providers are equal. Semi Analysis has some great articles explaining why.
Pricing -- exactly where shortages show up in a free market economy.
what I'm wondering: is there a shortage of compute in the current installed base? Why isn't the installed base of compute increasing drastically in price, and/or forcing me to wait in a queue to use it?
like, I can rent a VM from AWS for the same price as 5 years ago...
either AI achieves hyper-efficiency and triggers systematic economic demand destruction due to unemployment, or it underperforms and sparks a massive tech-sector asset bubble burst
If AI is as good as valuations - it crashes. If AI isn’t as good, it crashes. all eggs in one basket economy.
Every future leads to a massive financial shitstorm
AI is priced in as a technology to end all markets. The last bull market you will ever see.
It’s a miracle markets didn’t notice this “damned if you succeed damned if you don’t” type of inevitable checkmate
I have been building a bunker
They learned from 2008 that Too Big Too Fail(tm) means the government bails you out.
This was the problem with not OBLITERATING the banks in 2008. Had the US government taken over the banks and wiped the investors, there would be more pushback right now.
It's currently larger than the 1880s railroad boom.
"Overbuilding and speculative debt ultimately culminated in the Panic of 1893, plunging the nation into a severe economic depression."
"During the 1880s, private railroad enterprises represented over 60% of the total capitalization of the entire U.S. stock market, peaking at 63% in 1881 and 1885."
But as a % of total GDP, you are correct.
In terms of infra spending as a percent of GDP, 1880s railroads were at 5-6% at peak while AI is estimated to reach 3+% in a year or two.
Since HW specifics live mostly in the kernel layer (Pallas or CUDA), I imagine the Jax vs PyTorch question is more of a style preference now?
"No, but I have seen the house that the project built, and it is terrific." ~ some Google director
Is a pretty successful open source NVR that used corals when it launched (and many people still use them with it)
"OpenTPU: Open-Source Reimplementation of Google Tensor Processing Unit (TPU)" https://news.ycombinator.com/item?id=44111452
From https://news.ycombinator.com/item?id=45848676 :
> How does Cerebras WSE-3 with 44GB of 'L2' on-chip SRAM compare to Google's TPUs, Tesla's TPUs, NorthPole, Groq LPU, Tenstorrent's, and AMD's NPU designs?
There are TPUs made out of silicon carbide with standard fab.
There is in-flight tensor processing with photons, but that's not quite a TPU.
In that sense AMD is still better positioned to give NVidia a run for their money but given the family connection and AMD's underdog status I am not very hopeful that we'll see meaningful competition in inference hardware any time soon.
NVidia is a much harder nut to crack than OpenAI or Anthropic. They are easily a decade ahead of viable competition. Check out the GB300 if you don't believe that.
High bandwidth flash will be here soon, I bet someone makes an economical competitor thats not quite as fast but much cheaper.
Is this somehow Google's answer to today's best DeepSeek open-sourcing inferencing engine?
Personally the question is if they are "saving" other projects or news so they can be released in case a competitor makes a move? Who knows if this was ready days or weeks ago and were waiting on someone to drop some news, so they can shadow those news.