amazing to think there was a time (literally not too long ago) where you would throw 80 GB of HBM onto a chip and turn 90% of it off
cromka 23 hours ago [-]
Those were binned cards, though.
octoberfranklin 19 hours ago [-]
Binned for compute failures, not memory failures.
That's why this works. The yield on those HBM stacks is excellent. The binned cards had failures in the logic die, and hyperscalers want uniformity in their clusters. That's why the mining market took the downbinned cards.
londons_explore 16 hours ago [-]
Do you not test the logic die before mounting the memory on top?
octoberfranklin 9 hours ago [-]
Some testing can be done with a wafer probe, but other tests require stressing the die -- high current, heat output -- so it has to be in-package to do those tests.
Mounting the HBM on the interposer has to happen before putting the interposer in the package.
So it's the "must be in-package" yield tests that drive this.
monster_truck 12 hours ago [-]
back then, no
qurren 22 hours ago [-]
I'm wondering if this is actually the case or whether they were binned for defects in the HBM
kmike84 21 hours ago [-]
Anecdotically, 64GB works extremely reliably on 8GB cards after unlock.
It is much more of "let's sell cards to crypto people, but nerf them, to not harm our enterprise margins", not "we've got a ton of subpar hardware, let's to something with it".
londons_explore 16 hours ago [-]
crypto was a real boom... I could imagine these cards were already in production when the crypto guys came along willing to give lots of $$$ for them, but wanting really short lead times...
legitronics 11 hours ago [-]
I have a grey market a100 that had hbm errors but the card used some of its spare rows to remap. I believe they introduced this with ampere, and I was a little annoyed that my “new” card had these but I got it so cheap I didn’t really care. It was 7k usd (vs 9k and 10k) and I remember being unsure if it was worth it… None of them needed to remap any more in they years I’ve had them but I think I got that one so cheap because of it. The most expensive one came from the best rated seller and appears to be an engineering sample but has passed every test I’ve thrown at it, the only thing “wrong” with it is that I can set the power limit to whatever I want? This was back when b100 was about to be formally announced and before the pricing so eBay sellers were trying to unload them.
tencentshill 19 hours ago [-]
Sam Altman is probably buying them all up now
rkagerer 3 hours ago [-]
If someone wants to build a local AI GPU farm today, should they go out and buy a bunch of these? Or has the price already gone up too much to make it a good deal and is the "yield" you'd get (from things like bad segments of RAM, unreliability of the unlocks, SM lottery, etc.) too inconsistent?
kmike84 1 hours ago [-]
The "yield" is good enough, and the price reflects it. The unlock to 64Gb + PCIe Gen2 was there for months, this one is just a step further.
rootsudo 3 hours ago [-]
Too late to the game, go look at eBay and such,
nomorewords 2 hours ago [-]
A month or two back I saw available 5090s at 5k euro and thought that is robbery. Now there aren't any and third party sellers (none of the big chains have them) list at 10k plus
ketzu 12 minutes ago [-]
This made me check how much the 800€ AMD card I got last year is now: A) barely available, b) 2000€+
Wow.
notenchange 21 hours ago [-]
This is great. I’m actually trying to unlock the 80G in a CMP 170HX the software way for the last 2 weeks (still no luck).
I just don’t know if I’m brave enough (and knowledgeable enough) to go this blown fuses way. Seems a lot more risky.
P.S. I now see this is for the 8GB model, which is insane. The 10G one is the one that until now was proven to have 80G, but never fully accessible through CUDA (it was accessible using CPU memory copy though - much slower).
roger_ 17 hours ago [-]
Was the 80GB actually proven on the 10GB model? I thought it was a mirror of the first 40GB.
myself248 18 hours ago [-]
Instead of putting the host to ACPI S3 state, could an external PCIe riser be used with a separate power supply?
Seems like that would considerably speed up the process, by letting the host continue to run while only the target cycles.
amelius 1 days ago [-]
But what if nvidia used those fuses because an A100 had too many fabrication faults so it could only be a cheaper mining card?
(which is the more likely scenario because they can make much more money on enterprise/datacenter products, see DRAM)
puzzlingcaptcha 20 hours ago [-]
The article mentions that some die areas are factory-marked as defective and have to be "re-disabled" after the unlock. Ctrl-F _DEFECTIVE
stingraycharles 1 days ago [-]
Isn’t this how these things are always done? Same with number of cores in a CPU, the higher core and lower core come from the same fabrication process, it’s just that the lower core CPU had more faulty cores that are disabled.
mmo93 1 days ago [-]
Depends a lot on when its from in the product cycle, the yield at the end of production runs are usually so good that it turns into artificial product segmentation
BeeOnRope 1 days ago [-]
There's an aspect of that, but the primary reason for differences between various CPUs and GPUs which have the same underlying chip is product segmentation.
amelius 23 hours ago [-]
But nowadays they can make much more money with enterprise targeted products.
Which is why I think these devices are more likely to be faulty.
lukan 23 hours ago [-]
Nowdays - but this was a mining card, meaning bought for the crypto boom and before AI.
gabrielhidasy 20 hours ago [-]
The author mentions about 90% yield on getting the full 80GB of memory (in a limited 10 card sample).
1 days ago [-]
sidewndr46 23 hours ago [-]
Historically, there are products with high enough yield rates that it makes it irrelevant. See ATI Radeon 9500 Pro. A two byte patch in the firmware gives you an ATI Radeon 9800 Pro.
hypercube33 13 hours ago [-]
Used to be now we have chiplets. The defective chips ship as some weird OEM part usually now with parts disabled. Example: Ryzen 5 3500
I don't know what Intel is doing as I quit keeping up with them. I do miss the core unlocked boards that you'd put a 2 core cpu and maybe win the lottery on and get all 4 cores.
userbinator 10 hours ago [-]
This is why, for a while, AMD was selling 3-core CPUs. They were quads with a bad core, which could sometimes be unlocked.
amelius 1 days ago [-]
Yes, that's why I question the approach here.
Testing hardware is a complicated matter (you can do a PhD on the topic), and nvidia probably does a better job at it than a hacker in their basement who thinks they can upgrade their GPU for free.
On the other hand ... perhaps a few bit errors is not a problem for AI workloads ... but they could also crash your system ...
swiftcoder 17 hours ago [-]
The good news is that Nvidia did test the hardware, and left the "defective" masks right there in the fuses for us to read. If you have a card that has been fuse-segmented well below its defective part count, you have a product segmentation product on your hands.
jamesnorden 41 minutes ago [-]
>a hacker in their basement who thinks they can upgrade their GPU for free
Except he literally did.
LoganDark 16 hours ago [-]
Sometimes people order more lower-tier chips than there are defects, and then you get fully flawless silicon with some disablement.
sva_ 23 hours ago [-]
They say ECC works (?) so you would know.
I recently had a faulty A100, for some reason the provider turned off ECC on that card, so it silently failed and gave junk results... Took me a minute to figure out.
fn-mote 19 hours ago [-]
How did you ever figure this out?
sva_ 19 hours ago [-]
The weights loaded fine, but as I was training the loss made no sense. So I wrote some basic matmul test.
the8472 1 days ago [-]
Some applications may be more fault-tolerant than others though. If an LLM glitches you can reinitialize it, maybe reshuffle the allocations a bit and it'll be fine?
superxpro12 22 hours ago [-]
we call that dithering in the mechanical and analog worlds
b112 23 hours ago [-]
The sheer recklessness on display here. Everyone knows random bits is how something becomes sentient! You'll kill us all!
wil421 3 hours ago [-]
It would be hilarious if it became sentient but locked in an inferior GPU.
1 days ago [-]
phelix001 1 days ago [-]
They definitely did, but maybe you can get some good results running your own tests and determining if they really ARE bad or not, or they just overly tagged them out.
bob1029 22 hours ago [-]
HBM is so expensive that I doubt you would find significant gains on average. They are already yieldmaxxing like crazy.
nom 1 days ago [-]
it appears the opposite is true and that's why it's faster than an A100, which is optimized for yield and allows more defects
1 days ago [-]
roger_ 1 days ago [-]
Glad I jumped on these cards, excited to give this a try!
Hard to believe $1k was a deal a few months ago (and $200 before then).
gravypod 1 days ago [-]
I really wish I did. I told a friend about how I bet an unlock would come and decided to buy them when I heard about an unlock.
The highs and lows of my predictive capabilities
roger_ 1 days ago [-]
I usually miss out on these things, but I'd been monitoring the eBay prices rising for inferior cards and figured this was a good bet.
Annoying thing is finding a good motherboard to put multiple blower cards in. Mining rigs are a steal but PCIe x1 limitation is annoying. Got a bunch of Volta cards too sitting around until I figure out where to put them (and the performance per watt sucks next to the 170).
octoberfranklin 19 hours ago [-]
Supermicro chassis. Thank me later.
ForOldHack 18 hours ago [-]
Um, I should thank you right now, for that useful advice. Thanks.
numpad0 17 hours ago [-]
> I told a friend about how I bet an unlock would come and decided to buy them when I heard about an unlock.
What an idiotic idea. The price was low because the unlock didn't exist. They won't be selling at the price you weren't buying at after the unlock was discovered! Even 8GB would have been useful if you wpuld only waited for Strata to become a thing.
I'm not saying this to you, I'm trying to punch myself in January knowing about possible 64GB rumor and just telling people about it just like you, into retroactively buying it all by posting it here. Just why, me. This has been a great way to be salty about it for a whole year...
SillyUsername 1 days ago [-]
I just wish they had something like this for older Tesla p100s, p4s...
Considering how unobtainium GPUs have become nvidia could get a PR win by publishing unlocks for all their outdated hardware. ... people are figuring it out anyways, might as well embrace it.
ForOldHack 18 hours ago [-]
Unlocks and VRam upgrades, like VRam sim slots, and a VRom slot that ran Linux, had its own GUI for reconfiguration. Like old 68k macs that have a selectable os in the boot roms.
moffkalast 15 hours ago [-]
And skip the opportunity to sell them overpriced new cards? This is Nvidia we're talking about.
userbinator 10 hours ago [-]
The people doing this would likely not be buying new ones anyway.
dylanger 20 hours ago [-]
Great usage of glitching!
octoberfranklin 19 hours ago [-]
there are several functions where blowing a fuse on an NVIDIA die can grant you more access, not less. In a properly secured design, a blown fuse should always induce a more restrictive state, because a blown fuse cannot be unblown in an economically feasible way.
Not really.
If the fuses are for security, sure. But these aren't.
These fuses are for economic extraction. Doing it the way NVIDIA did it is a safety feature: it ensures that blowing a fuse will never destroy economic value -- it will never turn a card that was previously able to do something correctly into a card that can no longer do that thing correctly.
nomel 19 hours ago [-]
No, that's absolutely not the perspective of the bean counters at hardware companies. The fuses are also for economic security. If someone wants more features, you make sure that they paid for them. You don't give away higher tier SKU functionality for free. There was probably some cost/engineering justification for it, possibly like you say! Maybe too high of yield. This will almost certainly be removed in the next revision, as have all the many other similar cases before it.
For those unaware, product tiers come naturally with IC's, by disabling physical sections of the chip that can't work due to inevitable manufacturing defects. You don't throw defective chips away, you layout in a way where you can physically cut off those defective sections of the chip, with a saw and/or with a fuse, in a way that minimizes lost sales. This works great in the beginning, when your yields are low and you're still dialing in the manufacturing process. But, as yields increase, you might not have enough of the lower tier chips to meet demand. So you, just fuse otherwise good chips. Chips are cheap, after all! New revisions are not cheap, so features/bugs like this can remain around for a while.
In both IC companies I worked for, had around 6 ICs. If you opened any of them up, you would see the exact same die in every one. After a couple years of great yield, you would then find half missing in the lowest tier.
swinglock 19 hours ago [-]
Though if it enables hardware that wasn't enabled because of binning, making the card unstable, can you go back you go back to having a working card?
mitxela 23 hours ago [-]
It's utterly crazy that anyone outside Nvidia knows any of this, much less all of it. Has there been a years long public reverse engineering effort that I didn't hear about until now?
kmike84 22 hours ago [-]
Yes, there actually was a years long public reverse engineering effort; since 2023 there were some academic papers, solving parts of it. It has sped up a lot recently, probably due to accumulated knowledge, and due to AI use.
userbinator 10 hours ago [-]
Likely due to Nvidia-powered AI use.
AnthonBerg 3 hours ago [-]
beautiful!
Klein-bottle-shaped leopard's eating face
octoberfranklin 19 hours ago [-]
If you think about it, the effort is economically incentivized.
Buy a shitton of these things, spend a year working on the unlock. If you find one, sell the shitton at 5x higher price (which is what happened).
mr_roboto 23 hours ago [-]
Selling anything that has been purposely hamstrung to limit available capabilities for market reasons seems morally repugnant beyond words. I understand walling off die defects but if it is functional that is disgusting.
toast0 22 hours ago [-]
I understand your position, but as someone who prefers the lower end products (and the lower end prices!), this policy would be detrimental for me.
At the beginning of a product, when yields are bad, I could get a low tier chip at a good price, but then as yields get better, low tier chips wouldn't be available and I no longer have a low cost option. Presumably higher yields would lead to reduced prices on the higher tier chips, but I don't think they'd get down to where the low tier chips were.
Another alternative is that the maker only ever sells chips that test successfully as high tier, but I think it's repugnant to not make use of parts with defects if the defects can be gated off.
cowlevel 16 hours ago [-]
I think the idea is that as technology improves, you can get better technology for the same price. That's what the system of capitalism promised to us.
toast0 13 hours ago [-]
That does tend to happen... an 8 core Zen 5 (9700X) costs about the same today as the 6 core Zen 5 (9600X) did two years ago. But even if AMD doesn't have any defective 9700X to disable cores and sell as 9600X, they still want to attract the value buyer and so they'll take some perfectly fine chips, disable two cores and sell them as 9600X for less than the 9700X. They lose some margin, but they make it up in volume as we were promised.
tintor 22 hours ago [-]
When yields get better, manufacturer can just produce the lower tier chip directly.
toast0 21 hours ago [-]
This wouldn't really work for AMD processors... They could design and fab a 6-core chiplet, but it would need to be the same size as the 8-core chiplets so it would interface to the same carriers. If the footprint is the same, the fab cost is roughly the same, so spending the design time on that doesn't make sense. If they build separate carriers, that's more design time and inventory management and it may well be more expensive than just building 8-cores and disabling two.
That said, with Zen6 having 12 cores per chiplet, I wouldn't be surprised if they eventually end up designing a smaller chiplet with 6 cores and a dedicated small chiplet carrier. Or push 6 core SKUs to the APU side that doesn't use chiplets, Zen5 APUs already have specialized masks for some of their SKUs.
WarmWash 22 hours ago [-]
They still wouldn't because economies of scale make it cheapest to only build one model.
tintor 15 hours ago [-]
What economies of scale are you referring to?
Each wafer is printed separately. They don't build new custom production line for each new design.
bethekidyouwant 21 hours ago [-]
Its not economies of scale. Why or how would you even etch a worse pattern on the same fabrication node? They don’t even have a mask for the worse version its literally just the same pattern with more defects.
WarmWash 21 hours ago [-]
They only make one design because of economies of scale. I think there might have been some confusion here
cowlevel 16 hours ago [-]
Maybe you can get more of them per wafer because they're smaller?
bethekidyouwant 14 hours ago [-]
Yes but that’s a different thing those smaller chips also get binned based on defects
deaton 19 hours ago [-]
No, they can't. The lower tier chips exist pretty much entirely because they can be easily converted from the higher tier chips that are defective. Purpose-building a lower tier chip would be a ludicrously expensive thing to do, or else they'd already be doing that.
LoganDark 16 hours ago [-]
A fun example of this is Apple, who used to build three dies for each M-series generation (base, Pro, Max) but now build just two (base, Pro/Max). You'd think one of the most bespoke companies on the planet would not be moving to fewer dies unless there were very compelling reasons to do so.
arjie 22 hours ago [-]
This is one of the reasons a market is superior to centralized taste-setters. Despite the fact that both a seller and buyer can agree on a price for a less capable product, taste-makers often have these strange neuroses about what they should be paying for.
All sorts of reasonable interactions are then policed by people with visceral reactions to normal activity.
rustcleaner 12 hours ago [-]
There ought to be a law which stipulates that these segments are "soft," for the purpose of selling a working but partially defective product (and segmented due to that defect), and permit customers to undo the segmentation at their own risk. DMCA should not protect this practice IMO.
Aurornis 22 hours ago [-]
You’re going to be very disappointed to discover that this is common practice for many technology products. It’s cheaper and less risky to design a small number of things and then segment their capabilities than it is to make custom things for every different market segment.
It’s also useful for binning: These lower spec locked products can have defects in the unused parts, like flakey memory regions or channels. Not every CMP170 can be fully usable after unlocks.
The alternative to this market segmentation isn’t “Everyone gets the whole thing for the lowest price”. It’s that the lower price version doesn’t exist at all, or if it does it’s a lot more expensive.
judge2020 23 hours ago [-]
As this comment claims[0], it does seem like eventually it gets to where what you're saying is true, in that it's basically artificial segmentation.
With a bit of nuance of understanding it's not that wild or egregious, and no one is really getting cheated.
Basically the high price of the fully unlocked models is what makes it possible to sell artificially limited models at the lower price point (while also offering the ability to possibly sell lower binned units). Basically you can gas the profit on the high-end to drive down the cost on the low end, but obviously this only works if you have a distinct High and Low end.
People think that if we banned this practice it would lead to the top shelf models being sold at the bottom shelf price. But that's just naive market confusion. A ban would just mean that the top shelf unit at a higher price and the lower shelf unit at a higher price would be available.
There is no solve to the "problem" that results in "High shelf unit being sold at lower cost", in fact the thing you are complaining about is already the closest to that ideal, as the manufacturer can leverage economies of scale to drive down costs across the product line.
deaton 19 hours ago [-]
It's really no different. They are selling you a product with X advertised specs, and indeed the specs you can access as the customer (at least out of the box) are those exact advertised specs.
IncreasePosts 21 hours ago [-]
Imagine a world without this, where nothing was hamstrung and there was only one price point instead of a hamstrung low price and a nonhamstrung high price.
That one price would be somewhere between the low price and the high price. So, the world you want to live in would be one where "the poor" who currently pay the low price would need to pay more. And "the rich" would get a discount from their high price they pay currently. That seems morally repugnant beyond words.
Grimeton 18 hours ago [-]
Best not to imagine because they're both sold way past the manufacturing cost and I'm not under the impression that nvidia puts all the money back into development.
At least not the development of GPUs.
IncreasePosts 17 hours ago [-]
The margins are irrelevant here. If Nvidia sold them for the exact cost of manufacturing, people would just scoop them up and re-sell them to the highest bidder. So in the case of 0% margin, some useless middle-man would get to profit.
remindmi16 16 hours ago [-]
[flagged]
Savlume 1 days ago [-]
[flagged]
samlinnfer 1 days ago [-]
Before you get too excited about the CMP 170HX, the prices have already skyrocketed and that card doesn't have a cooler on it. It is designed for a server cooler that sounds like a jet engine. Also most of the 80GB vram unlocks have bad RAM and that is the reason they were locked in the first place, otherwise they would have became A100s.
Anyways, that's what I tell myself as I taste my bowl of sour grapes.
Yeah the fan noise is only noticeable when I push the cards to ~180W, otherwise they idle pretty quiet.
How are you dealing with the RAM limit (32GB)? I had upgraded mine to an i7 but it was unstable so I’m back to the stock Celeron class CPU.
segmondy 17 hours ago [-]
I'm not using octominer MB or CPU. That MB on my octominer case is running everything, it's an epyc cpu with 512gb of dram. octominer is also pcie limited, i get faster pcie lane with this.
NoidFonsense 17 hours ago [-]
Here is where I get to smugpost about how I bought a ton for $800 each, against the disgresion of my friends and claude who did not believe in the exploit...
After which I sold most of them for $2000-$2600 (I believed the panic about the new mac studio launch) instead of holding them to sell for like $5k and making my annual income in a week. And also I failed to do the capacitor mod after buying a couple hundred in soldering equipment. But at least I sold most of them for monero, and that's gone up since and allowed me to avoid massive ebay fees.
The moral of the story here is that you can fantisize about getting in at the bottom, but even if you did you may not have rode it to the top. Plenty of people bought bitcoin when it was half a cent and sold at the local maxima to net $30k and spent it on a used corrolla.
I think my story may make your grapes less sour.
Edit: I knew at the time of purchase that the 10GB->80GB unlocks were problematic but the 8GB->64GB cards were solid. The pricing reflects this, so thats just skill issue.
roger_ 1 days ago [-]
I sensed the sour grapes :D
Bad RAM isn't a huge issue as you can flag and ignore the relevant regions and be down a few MB.
r_lee 24 hours ago [-]
I remember looking at research n thinking if they'd be useful for AI with their integer ops
I even looked into blog posts of using them or unlocking them somehow but I never thought to think that they actually had the full HBM memory and all just fused away??? wow
tintor 22 hours ago [-]
Finally, I can download more VRAM!
tclancy 21 hours ago [-]
Just don’t download a car.
ForOldHack 18 hours ago [-]
You wouldn't want to download a car, you would want to download a dozen cinnimon rolls. How ya doin?
RemingtonDavies 21 hours ago [-]
[dead]
edg5000 9 hours ago [-]
Why on earth would you disable so much HBM from the factory. That's incredibly wasteful. No wonder these things cost an arm and a leg. It was already clear the money was going to the HBM primarily. Either the post is wrong or this is actually rediculous.
solyin 8 hours ago [-]
Human psychology and money. If I offer a small, medium, and large of something, you're more likely to buy something rather than if I only offer one thing and you must like that one thing.
richlow 16 hours ago [-]
Firstly; this is awesome!
Secondly you never spoke about the squeaky bum moment of actually making the change. Crikey that would have been a long moment.
userbinator 10 hours ago [-]
Unlocking regular GPUs into Quadro GPUS via resoldering some resistors was a thing long ago, so I'm not surprised this is possible; but that didn't unlock any extra VRAM.
These findings were disclosed to NVIDIA’s product security team ahead of the publication of this article.
WTF. Does the boot taste that good?
Kevin_Flynn 1 days ago [-]
The ebay auction headlines have already rewritten themselves.
EmX_3000 15 hours ago [-]
NVIDIA driver "fix" in 3..2..1...
mschuster91 24 hours ago [-]
> In one of the most extreme manifestations of product segmentation, the CMP 170HX, an Ethereum mining card with nominally 8 or 10 GB of application-exposed framebuffer capacity, actually comes with, respectively, ~80 and ~40 GB of HBM onboard!
wtf, it seems RAM truly was too cheap back then...
pwg 23 hours ago [-]
The marginal cost to fabricate the GPU die with the full ram allotment, and then disable a portion for 1) salvaging otherwise useless die's with partly bad ram for lower ram use cards; or 2) market segmentation; is tiny compared to the marginal cost of creating X different masks and fabricating X different die's, one specific to each market segment/ram size amount.
In addition, fabricating the same die for every SKU point in the product line means the maker does not have to guess how many version X vs. version Y vs. version Z dies to fabricate. The just have one fabricated. Then the dies become SKU version X, Y or Z as demand requires during the card assembly process.
jeroenhd 23 hours ago [-]
The card was initially priced at over 4 grand and it had its capabilities severely reduced to make it not compete with real GPUs.
Plus: most of those cards probably have broken RAM chips anyway. Throwing them out would be a waste, limiting trash chips to a small amount of RAM that does work turns recycling costs into a product that can be sold for money.
throwaway_new_1 23 hours ago [-]
Scaling according to demand is way too hard when it comes to silicon.
mitxela 23 hours ago [-]
Nvidia might've been planning to eventually buy them back from the miners, or sell software RAM unlocks.
dreamcompiler 20 hours ago [-]
Imagine if cars were like this: "We will sell you this Ferrari for $2K and it will perform like a Chevy Vega until you give us another $200K, which will unlock the hardware."
Oh wait, cars are like this now.
ForOldHack 18 hours ago [-]
Swear to God, this is absolutely true. Had a friend who had a RAV4. No tow package for low end torq. So... waited a few months, spotted a programmer on Craigslist, said it had tow mode. It had 4 more models too. Super tow, super economy and stock. Buyer was beside themselves, learned how well they worked, and would switch to super eco on one ramps. ( really a bad idea, so they stopped...)
Hacking is a good thing.
ForHackernews 23 hours ago [-]
>an Ethereum mining card
Wait, so does this unlock it to actually do something useful, or just "mine" marginally more shas?
coffeebeqn 23 hours ago [-]
Eth isn’t mined anymore so that wouldn’t be particularly useful
KiwiJohnno 12 hours ago [-]
You can run a local LLM on it, with performance you would otherwise need to spend $20k on hardware for.
ipsum2 18 hours ago [-]
You can use it for machine learning, or anything a normal A100 is used for.
andai 3 hours ago [-]
> a blown fuse cannot be unblown in an economically feasible way
Isn't it like a bit of solder? Or is the problem that the solder would have to be a nanometer across?
arghwhat 3 hours ago [-]
efuses use electromigration whereby a higher than normal voltage causes metal to migrate away from the junction, not a wire getting burnt through like a regular sacrificial fuse.
If you wanted to patch a die the only real means is through a FIB (Focused Ion Beam) workstation, and even then you need the die structure to allow it (you have to mill your way into the chip and can't have any functional layers in between you and the patch you want to makey), and even when doable it would be an insanely expensive, time-consuming and likely fragile project.
numpad0 3 hours ago [-]
A "fuse" in this context are transistors and bit-addressable RAM with no erase lines in chip, not physical fuses. They can't be unblown without a Star Trek replicator, or at least like a high powered electron microscope or something.
gwbas1c 23 hours ago [-]
> In a properly secured design, a blown fuse should always induce a more restrictive state, because a blown fuse cannot be unblown in an economically feasible way.
I disagree: If there's any risk that the fuse will blow with normal usage, then a blown fuse means a defective product.
bluGill 23 hours ago [-]
These are not safety fuses. They will never, in normal use, blow. These are fuses used only in a logic circuit. There is a way that the factory can make the fuse blow and then the circuit code or whatever can check is that fuse blowing and if so it will not do something.
cowlevel 16 hours ago [-]
"fuse" here refers to one-time-programmable memory, not to a safety current limiting device
low_tech_punk 23 hours ago [-]
What if the author turn out to be Alice Huang?
Joke aside, good investigation and blog debut, kate!
Rendered at 13:23:34 GMT+0000 (Coordinated Universal Time) with Vercel.
That's why this works. The yield on those HBM stacks is excellent. The binned cards had failures in the logic die, and hyperscalers want uniformity in their clusters. That's why the mining market took the downbinned cards.
Mounting the HBM on the interposer has to happen before putting the interposer in the package.
So it's the "must be in-package" yield tests that drive this.
It is much more of "let's sell cards to crypto people, but nerf them, to not harm our enterprise margins", not "we've got a ton of subpar hardware, let's to something with it".
Wow.
I just don’t know if I’m brave enough (and knowledgeable enough) to go this blown fuses way. Seems a lot more risky.
P.S. I now see this is for the 8GB model, which is insane. The 10G one is the one that until now was proven to have 80G, but never fully accessible through CUDA (it was accessible using CPU memory copy though - much slower).
Seems like that would considerably speed up the process, by letting the host continue to run while only the target cycles.
(which is the more likely scenario because they can make much more money on enterprise/datacenter products, see DRAM)
Which is why I think these devices are more likely to be faulty.
I don't know what Intel is doing as I quit keeping up with them. I do miss the core unlocked boards that you'd put a 2 core cpu and maybe win the lottery on and get all 4 cores.
Testing hardware is a complicated matter (you can do a PhD on the topic), and nvidia probably does a better job at it than a hacker in their basement who thinks they can upgrade their GPU for free.
On the other hand ... perhaps a few bit errors is not a problem for AI workloads ... but they could also crash your system ...
Except he literally did.
I recently had a faulty A100, for some reason the provider turned off ECC on that card, so it silently failed and gave junk results... Took me a minute to figure out.
Hard to believe $1k was a deal a few months ago (and $200 before then).
The highs and lows of my predictive capabilities
Annoying thing is finding a good motherboard to put multiple blower cards in. Mining rigs are a steal but PCIe x1 limitation is annoying. Got a bunch of Volta cards too sitting around until I figure out where to put them (and the performance per watt sucks next to the 170).
What an idiotic idea. The price was low because the unlock didn't exist. They won't be selling at the price you weren't buying at after the unlock was discovered! Even 8GB would have been useful if you wpuld only waited for Strata to become a thing.
I'm not saying this to you, I'm trying to punch myself in January knowing about possible 64GB rumor and just telling people about it just like you, into retroactively buying it all by posting it here. Just why, me. This has been a great way to be salty about it for a whole year...
Not really.
If the fuses are for security, sure. But these aren't.
These fuses are for economic extraction. Doing it the way NVIDIA did it is a safety feature: it ensures that blowing a fuse will never destroy economic value -- it will never turn a card that was previously able to do something correctly into a card that can no longer do that thing correctly.
For those unaware, product tiers come naturally with IC's, by disabling physical sections of the chip that can't work due to inevitable manufacturing defects. You don't throw defective chips away, you layout in a way where you can physically cut off those defective sections of the chip, with a saw and/or with a fuse, in a way that minimizes lost sales. This works great in the beginning, when your yields are low and you're still dialing in the manufacturing process. But, as yields increase, you might not have enough of the lower tier chips to meet demand. So you, just fuse otherwise good chips. Chips are cheap, after all! New revisions are not cheap, so features/bugs like this can remain around for a while.
In both IC companies I worked for, had around 6 ICs. If you opened any of them up, you would see the exact same die in every one. After a couple years of great yield, you would then find half missing in the lowest tier.
Klein-bottle-shaped leopard's eating face
Buy a shitton of these things, spend a year working on the unlock. If you find one, sell the shitton at 5x higher price (which is what happened).
At the beginning of a product, when yields are bad, I could get a low tier chip at a good price, but then as yields get better, low tier chips wouldn't be available and I no longer have a low cost option. Presumably higher yields would lead to reduced prices on the higher tier chips, but I don't think they'd get down to where the low tier chips were.
Another alternative is that the maker only ever sells chips that test successfully as high tier, but I think it's repugnant to not make use of parts with defects if the defects can be gated off.
That said, with Zen6 having 12 cores per chiplet, I wouldn't be surprised if they eventually end up designing a smaller chiplet with 6 cores and a dedicated small chiplet carrier. Or push 6 core SKUs to the APU side that doesn't use chiplets, Zen5 APUs already have specialized masks for some of their SKUs.
Each wafer is printed separately. They don't build new custom production line for each new design.
All sorts of reasonable interactions are then policed by people with visceral reactions to normal activity.
It’s also useful for binning: These lower spec locked products can have defects in the unused parts, like flakey memory regions or channels. Not every CMP170 can be fully usable after unlocks.
The alternative to this market segmentation isn’t “Everyone gets the whole thing for the lowest price”. It’s that the lower price version doesn’t exist at all, or if it does it’s a lot more expensive.
So I guess the next step is to stop selling the lower-binned card? https://www.tomshardware.com/pc-components/gpus/nvidia-repor...
0: https://news.ycombinator.com/item?id=50452839
Basically the high price of the fully unlocked models is what makes it possible to sell artificially limited models at the lower price point (while also offering the ability to possibly sell lower binned units). Basically you can gas the profit on the high-end to drive down the cost on the low end, but obviously this only works if you have a distinct High and Low end.
People think that if we banned this practice it would lead to the top shelf models being sold at the bottom shelf price. But that's just naive market confusion. A ban would just mean that the top shelf unit at a higher price and the lower shelf unit at a higher price would be available.
There is no solve to the "problem" that results in "High shelf unit being sold at lower cost", in fact the thing you are complaining about is already the closest to that ideal, as the manufacturer can leverage economies of scale to drive down costs across the product line.
That one price would be somewhere between the low price and the high price. So, the world you want to live in would be one where "the poor" who currently pay the low price would need to pay more. And "the rich" would get a discount from their high price they pay currently. That seems morally repugnant beyond words.
At least not the development of GPUs.
Anyways, that's what I tell myself as I taste my bowl of sour grapes.
That said, 80G for something that you picked up for cheap could be an absolute steal these days.
Currently listening to the soothing drone of my $200 Octominer cooling my cards while I'm fine tuning a model (almost makes a good space heater).
https://www.reddit.com/r/LocalLLaMA/comments/1wjr59t/768gb_v...
How are you dealing with the RAM limit (32GB)? I had upgraded mine to an i7 but it was unstable so I’m back to the stock Celeron class CPU.
After which I sold most of them for $2000-$2600 (I believed the panic about the new mac studio launch) instead of holding them to sell for like $5k and making my annual income in a week. And also I failed to do the capacitor mod after buying a couple hundred in soldering equipment. But at least I sold most of them for monero, and that's gone up since and allowed me to avoid massive ebay fees.
The moral of the story here is that you can fantisize about getting in at the bottom, but even if you did you may not have rode it to the top. Plenty of people bought bitcoin when it was half a cent and sold at the local maxima to net $30k and spent it on a used corrolla.
I think my story may make your grapes less sour.
Edit: I knew at the time of purchase that the 10GB->80GB unlocks were problematic but the 8GB->64GB cards were solid. The pricing reflects this, so thats just skill issue.
Bad RAM isn't a huge issue as you can flag and ignore the relevant regions and be down a few MB.
I even looked into blog posts of using them or unlocking them somehow but I never thought to think that they actually had the full HBM memory and all just fused away??? wow
Secondly you never spoke about the squeaky bum moment of actually making the change. Crikey that would have been a long moment.
These findings were disclosed to NVIDIA’s product security team ahead of the publication of this article.
WTF. Does the boot taste that good?
wtf, it seems RAM truly was too cheap back then...
In addition, fabricating the same die for every SKU point in the product line means the maker does not have to guess how many version X vs. version Y vs. version Z dies to fabricate. The just have one fabricated. Then the dies become SKU version X, Y or Z as demand requires during the card assembly process.
Plus: most of those cards probably have broken RAM chips anyway. Throwing them out would be a waste, limiting trash chips to a small amount of RAM that does work turns recycling costs into a product that can be sold for money.
Oh wait, cars are like this now.
Hacking is a good thing.
Wait, so does this unlock it to actually do something useful, or just "mine" marginally more shas?
Isn't it like a bit of solder? Or is the problem that the solder would have to be a nanometer across?
If you wanted to patch a die the only real means is through a FIB (Focused Ion Beam) workstation, and even then you need the die structure to allow it (you have to mill your way into the chip and can't have any functional layers in between you and the patch you want to makey), and even when doable it would be an insanely expensive, time-consuming and likely fragile project.
I disagree: If there's any risk that the fuse will blow with normal usage, then a blown fuse means a defective product.
Joke aside, good investigation and blog debut, kate!