Chinese AI companies are free to use any Nvidia GPU they want as long as the GPU is not physically located within China. It is entirely consistent with US export controls for Chinese AI companies to remotely use GPUs located in Singapore or Malaysia.
“In facilities across the region, Megaspeed rents Nvidia chips to Chinese tech giants such as Alibaba, an arrangement that does not violate current US export control regulations.” See [1].
Literally my first thought. Noone can stop them from training in data centers that are located elsewhere. It's not like some military hardware that needs to be physically shipped to China for them to make use of it. In fact why would they bother with Chinese compute providers if those are limited because of GPU export controls? They knew this for a long time and probably have been knocking on the door of providers in other countries for a while. Heck, they could be training inside the US. They already distribute their open model weights from the US.
gpt5 23 hours ago [-]
The US definitely has the power to stop most of it. It just looks like the executive branch doesn’t yet have the legal authority to do it, but it’s in an advanced legislative state.
flowerlad 23 hours ago [-]
How are you going to monitor who is logging into a system, or who a job is being run for? If you want that level of control the machines should not be exported at all.
vikramkr 10 hours ago [-]
Datacenters are very large and not very mobile and full of very expensive export controlled equipment and training runs are large expensive contracts. This is not exactly a difficult context to enforce KYC rules in
cowlevel 15 hours ago [-]
Same way they do for Iran: if they catch you providing services to someone in Iran, they send you to jail for life. And they monitor all financial transactions so they'll notice.
graemep 14 hours ago [-]
They can do that to Americans. Harder for the rest of the world. How is the US going to monitor CIPS payments? How will they jail someone who never goes to the US? Just how much global jurisdiction is the US claiming?
throwaway2037 12 hours ago [-]
In practice, the key ways that the US exerts influence outside of its own borders: (1) Sanction the company, which means any other company that does business with that company can also be sanctioned. (2) If enough companies are evading sanctions in a single country or the govt is helping those companies, then threaten to cut the country from USD settlement with the Federal Reserve, e.g., no USD FX allowed. That pretty much screws most economies. The "long arm" of the US govt is very, very, very long!
To be clear: I am not saying any of this is fair (or legal); I am only explaining how it works.
cowlevel 13 hours ago [-]
How is the money going to end up in a US account via CIPS?
graemep 10 hours ago [-]
Why would it need to end up in a US account?
cowlevel 7 hours ago [-]
That's where Jensen Huang is
graemep 4 hours ago [-]
Does he own any data centres outside the US?
tremon 11 hours ago [-]
> How will they jail someone who never goes to the US?
The same way they "jail" UN-appointed judges that go against their masters' interests: de-platform them from all US-controlled payment and software platforms.
graemep 7 hours ago [-]
That is not jail.
if you are referring to people at the ICC, they are cut off from US services, but that is a long way from being in jail.
cowlevel 7 hours ago [-]
Can't travel, can't shop, can't have a cellphone or even a landline...
graemep 4 hours ago [-]
Yes, they can. They cannot buy things from Amazon or book through booking.com, as that French judge said, but they can do things that do not require American systems. Do you have any evidence they cannot have phones? What stops they shopping in person? Or booking directly with a hotel?
> He said: “We are going 30 years back in time. It’s like a time machine returning us to a pre-digital world,” he told France TV.
If you would be so kind as to read your own source, it answers your questions
awesome_dude 18 hours ago [-]
The same way that SAAS and IAAS (etc) providers do now for citizens of other countries under sanctions - IPs, Credit Cards
mahkeiro 17 hours ago [-]
So another company in Asia is going to execute the batch and send the result back to China. And you can add as many indirection levels as you like. This is going to be impossible to track.
vikramkr 9 hours ago [-]
You'd just invert it. Make it so you have to be approved under export controls as a customer like they were gonna do for mythos (and like how mythos sort of still works) and make it an allowlist instead of a denylist
awesome_dude 16 hours ago [-]
If that company is acting in defiance of sanctions, like any other company that acts as a middleman to circumvent sanctions, and it is caught, it will face severe consequences.
There's no gotcha here, companies the world over could just ignore any sanctions they wanted, and some try to, but they do so knowing they themselves are (potentially) breaking local and international laws.
You might have well asked "what if a company helps you by importing contraband instead of you doing it directly"
15 hours ago [-]
Qiu_Zhanxuan 15 hours ago [-]
None of them are breaking local and international laws by doing so, unless a country's ruling says other countries laws supersede their own
awesome_dude 14 hours ago [-]
One of the key reasons that a country like the USA can enforce sanctions is that it can tell other countries that if their companies/citizens/etc break the sanctions, then the USA will put restrictions on those countries ability to trade with the USA, which is why only a few countries will trade with Iran, North Korea, and Russia
nmfisher 10 hours ago [-]
At some point those countries will also tire of the USA trying to tell them what to do and will tell the USA politely to fuck off. This is basically what India did with US sanctions on Russia and Iran.
Given the USA has completely torched its reputation amongst allies in the space of a year, I would expect this to happen more often.
throwaway2037 12 hours ago [-]
Exactly right. The term I use is "economic network effect".
Does anyone know why does the US allows Canada to trade with Cuba? That one always stood out to me as a bit odd. My guess: Canada is the largest trade partner with the US, so the US probably has more to lose by sanctioning Canada.
cowlevel 15 hours ago [-]
Yes, that's called a treaty, or a trade agreement. Remember all the fuss about Investor/State Dispute Settlement clauses? Most countries have them with each other by now.
awesome_dude 14 hours ago [-]
To add to this, if the provider suspects that a customer is facilitating sanction busting, or could reasonably be expected to have foreseen it, that's enough for a discussion with lawyers...
LtWorf 15 hours ago [-]
Lol what consequences? Such shell companies can appear by the thousands every hour. Even if you ban 2 or 3 it doesn't matter.
awesome_dude 14 hours ago [-]
Funny, sanctions have successfully applied to countries such as Russia, Iran, etc for decades...
They do try workarounds, and when those are caught there are consequences
You can sit back down now.
Edit: Is there going to be more "laws can be broken so let's not bother" arguments, because that's lame and boring
graemep 14 hours ago [-]
> sanctions have successfully applied to countries such as Russia, Iran, etc for decades.
Largely because other countries are willing to go along with it, and to the extent they are willing to do so.
Even NATO countries import Russian gas. The UK rolled back sanctions on Russia to import diesel earlier this year. Most of the rest of the world did not impose effective sanctions and lots of people trade with them (including huge economies like China).
The rest of the world is to dependent on China to want to get involved. The US might succeed in the current administrations attempt to split the world into new cold war spheres of influence (because people are scared of China too) but its not going to be easy, it will be painful, and its not going to get many countries to completely agree to what the US wants.
LtWorf 12 hours ago [-]
> They do try workarounds, and when those are caught there are consequences
Lol. Yeh we can all see how well the sanctions worked on Russia and Iran. Thanks for proving my point!
awesome_dude 3 hours ago [-]
Again with the "laws are broken so let's not have them" argument.
petesergeant 19 hours ago [-]
You're going to make it the provider's problem, unleash the full wrath of the US Government on one or two who don't comply pour encourager les autres, and then it self-regulates(ish).
QuantumNomad_ 17 hours ago [-]
> pour encourager les autres
I was wondering why you wrote this part in French instead of English. Turns out it is a phrase that is sometimes loaned into English. It comes from the (French) writings of Voltaire.
How so? They would need to forbid nvidia to sell to anybody but US customers
vikramkr 9 hours ago [-]
Or our allies that we control. But in terms of how - remember the mythos launch? Im absolutely not saying it's necessarily a good idea, I'm just that it's not that hard if we want to do it.
LtWorf 15 hours ago [-]
And regulate all the owners of nvidia hardware. Including, somehow, those who aren't in the USA.
jjtheblunt 22 hours ago [-]
how would that work?
22 hours ago [-]
cowlevel 15 hours ago [-]
They're also free to use Nvidia GPUs physically located in China. However, customs and border enforcement will try to stop them from moving GPUs physically located outside China to physical locations inside China. However, if they can bypass that, they can use the GPU.
what_hn 15 hours ago [-]
This is the key statement. Lol
Thlom 17 hours ago [-]
Is there any international regulation that forces China to comply with arbitrary US export controls? If not I can't see why Chinese companies are not free to use any Nvidia GPU they can get their hands on period (How the US would then react towards China, the Chinese company and Nvidia is another question).
cowlevel 15 hours ago [-]
No there isn't. If they have an Nvidia GPU, then they have it
"A key player in getting Megaspeed off the ground was Alibaba, the Chinese e-commerce leader that has also been an aggressive player in AI, said people familiar with the negotiations at the time. The U.S. had begun imposing export controls on advanced chips in late 2022, preventing Nvidia from selling them directly to China over concerns they could boost China’s military and surveillance capabilities.
"Alibaba was looking for a partner that could help it access computing power from data centers stuffed with Nvidia chips located outside China, the people said."
hereme888 23 hours ago [-]
[flagged]
x313 1 days ago [-]
The author of this piece is a SWE-turned-influencer "China hawk" from OpenAI (i.e. the primary advocate of export bans along with Anthropic), and this isn't an academic paper despite the theming.
I have no idea whether this is legit or not (there's no link to any code), I'm mentioning this because all the comments here are assuming truth without verifying the contents.
cptcobalt 1 days ago [-]
This post is from Will Depue
The supposed “China Hawk” you mention, on google searching, seems to be David Perdue.
Is there a chance you are confusing similar yet different last names as the same person?
x313 1 days ago [-]
No, the names are correct. OpenAI has quite a few SWEs who spend a lot of time advocating for positions on social media.
iantpe 17 hours ago [-]
I don't understand. What difficulties would these Chinese companies face if they held their training overseas?
jonathanstrange 16 hours ago [-]
The US could impose secondary sanctions to close the loophole, or pressure these countries behind closed doors to not allow these data center uses. Since the only goal of the export restrictions is to hamper Chinese AI efforts, that would be a logical step. If there is one thing that is certain then it's that the current US administration doesn't care at all about open trade and international trade agreements.
modeless 24 hours ago [-]
The method seems sound and the findings correct. But I think many are missing that (as the research itself says) this doesn't prove any illegal behavior. It's perfectly legal for Chinese companies to train models on GPUs that they rent outside of China. It's also legal for companies to buy GPUs and install them in Malaysia with the intention of renting them to Chinese companies.
There isn't even an allegation that the Chinese companies lied about this. They simply didn't disclose which chips were used. In all cases where they did disclose the training chips, this research found the disclosures to be accurate.
adrian_b 13 hours ago [-]
The small print in TFA states the many limitations of their method, i.e. that it actually provides no evidence about which GPUs were really used for training, and much less about who was their owner and where they were located.
The method just strongly suggests that the initial values of the weights, before starting training, were generated with the NVIDIA cuRAND library on a Blackwell GPU (or on an AMD GPU, where the NVIDIA library is emulated for compatibility).
The training might have been done with thousands of GPUs of other types. It is plausible that the GPUs used for training or at least a part of them may have been of the same type with the one used for initializing the model, but it is not proven in any way.
A scenario that matches what is reported in the article, but in which there is only a single Blackwell GPU involved, is if the LLM was intialized on the workstation of some lead developer, who had the privilege to use the best GPU they had, and then it was transferred to a server cluster for training, where other kinds of GPUs were used.
Now that this has been published, it would be very simple in the future for any Chinese AI company to avoid this "fingerprinting", by just using a different PRNG, instead of cuRAND.
modeless 7 hours ago [-]
Yes it's simple to avoid this fingerprinting now that people know about it, but nobody knew about it before. I don't believe people are casually transferring snapshots between chip architectures without a really good reason, either. It's hard enough to coordinate these training runs without involving multiple chip architectures in a single run.
RobertoG 16 hours ago [-]
Does 'legal' an 'illegal' apply here? Surely, the USA legal system only apply in the USA?
modeless 8 hours ago [-]
'Legal' and 'illegal' apply to Nvidia or other US companies facilitating these activities. Perfectly legal for Nvidia to supply GPUs to be rented by Chinese companies, or for Oracle to build datacenters for Chinese companies to rent, as long as they are installed outside of China.
cowlevel 15 hours ago [-]
You'd think so but the USA is also the country that's most advanced at getting its legal system to apply outside of it. E.g. they shut down the International Criminal Court.
So not "shut down" but unsubstantiated reports that one country is in talks with the US about limiting its powers.
LtWorf 15 hours ago [-]
>USA is also the country that's most advanced at getting its legal system to apply outside of it
Yes it has nukes. However china also has nukes. That poses some legal limitation.
cowlevel 15 hours ago [-]
The USA didn't need nukes to get Venezuela.
m4rtink 9 hours ago [-]
Did they really "get" Venezuela ? What actually prevents them to start ignoring what US wants in a while ?
LtWorf 12 hours ago [-]
It needed them to avoid retaliation from the rest of the world.
sam0x17 24 hours ago [-]
it's funny because it convinces me that export controls don't work so why do them
tristanj 18 hours ago [-]
I used to think the same, that export controls didn't work, until I realized that China has leapfrogged the West on every technology that wasn't subject to export controls. And the only ones where they're still behind are the ones under export controls.
cowlevel 15 hours ago [-]
The purpose of a system is what it does, not what it doesn't do.
RobotToaster 24 hours ago [-]
It's like the war on drugs, just one more crackdown will fix everything this time.
chrisco255 18 hours ago [-]
Its more like the war on bugs. Just because you can't kill them all doesn't mean you shouldn't spray and lay pest control.
cwmoore 21 hours ago [-]
because they are bad
GenerWork 1 days ago [-]
It's insane how many articles have dropped in the past 24 hours that all have a similar tone. At this point it's hard to see this as anything other than a broad coordination from someone who wants to destroy Jensen/Nvidia.
dust42 15 hours ago [-]
There is a 1T IPO coming up. I'd be surprised if they didn't hand over plenty of bucks to PR companies, grass root campaigns, influencers and journalists.
epistasis 24 hours ago [-]
This is how the news works on a story of major importance. As details of corruption come out, they all get reported.
One bigger aspect of this is that it was all saved from being leaked until right after the election.
Somebody was playing partisan politics with national security.
One key question: did Trump recently sell NVDA? How much and when? He's known of this for more than a year.
NooneAtAll5 24 hours ago [-]
if we're going paranoid mode, on 9th of November Trump said that "leader of one large company connected to artificial intelligence" was almost sent to prison by Ministry of Justice due to wrongdoings connected to Ai, and this threat caused immediate correction by that company
m4rtink 9 hours ago [-]
I guess fingers crossed & good luck to them? After making consumer hardware unaffordable due to their greed I guess that is a reasonable outcome & karma at work.
octoberfranklin 20 hours ago [-]
the Deep State has entered the chat.
martinwright 22 hours ago [-]
I definitely care about this much less than it seems I'm supposed to.
I think the frontier models are definitely in a healthy dynamic where it seems the near-but-not-quite performance of open-weight models are what is pushing the most innovation and price competition from OpenAI, Anthropic and co
panicinducer 22 hours ago [-]
Some people seem to think that American sanctions and export controls are some form of absolute justice, and that any violation is an affront to humanity.
If GPUs are going to be priced out of consumer reach anyway, I would rather see the GPUs used to create open weight models than just to make these AI companies richer.
chrisco255 18 hours ago [-]
Well the problem is you can't prove they are only being used to create open weight models. You can't prove they aren't also creating closed weight models with military applications beyond what they've publicly open sourced.
tancop 17 hours ago [-]
Why should I care that China is making some super powered hacking model? We already know they do state sponsored APTs. It's just motivation to care about security and red team your server code.
Their government is not really the type to violently take over other countries and impose their laws. Taiwan is a special case because they see it as a rebel state on their own land. The rest of us are probably better off with China being the top superpower than Trump's America.
loneboat 19 hours ago [-]
> We recover traces of the models’ initial values, and then search for the hardware configurations and initial configurations that reproduce them
From my (very cursory, naive, layman's) glance at the summary, they're identifying and tracing some characteristics that specific hardware imparts into a model's initialized weights. But then concluding that this indicates the models were also "trained on" that same hardware.
But is that a valid assumption? Is it possible that a model was initialized on one system configuration, while the bulk of the training was performed on another?
In fact, it would seem reasonable for the "okay, I'm setting up this new model on my workstation, prepping it for training next month" pipeline step to be far removed from the "okay, now deploy the [initialized, but not yet trained] model across our giant fleet of beefy training machines".
culi 8 hours ago [-]
> The newest models we could read from Alibaba (Qwen), Zhipu (GLM), Tencent, Xiaomi and StepFun all read as Hopper-trained. Meituan says its LongCat-2.0 was trained entirely on “AI ASIC superpods,” and we found no NVIDIA fingerprint in the model weights it’s released.
Yeah makes sense. They are in transition. This article doesn't really tell us anything new (or interesting). There's nothing wrong w Chinese labs using GPUs that are already in the country. An export ban just means reduced import of new GPUs
basiccalendar74 1 days ago [-]
as the post mentions, the models could have been trained on 1. GPUs within China or 2. GPUs outside China for example datacenters in Singapore.
The 2nd option is perfectly legal and allowed by US government.
cowlevel 15 hours ago [-]
The 1st option is also perfectly legal and allowed by the US government. Only the export of GPUs is illegal, and only sometimes.
dcrazy 24 hours ago [-]
Is the U.S. government OK with this if the money to buy those GPUs and run inference on them comes from Chinese companies?
seanmcdirmid 23 hours ago [-]
yes. The USA hasn't force an embargo on China to other countries (e.g. by threatening to cut off Swift access). We aren't technically at war either, so its highly unlikely that the US would go that far.
What is surprising, I guess, is that the US allows GPUs to be exported at all to most countries. Since...as long as they are exported to a country that is friendly to china (which is basically most countries on the planet), China has access to GPUs on which to train with, making GPU purchasing restrictions mainly symbolic.
HDThoreaun 18 hours ago [-]
They dont need to threaten other countries, they can just pass a law saying that anyone anywhere in the world selling or renting gpus to chinese companies is committing a crime. Then they can sanction any company that does it anyway and that makes it a crime for any american company to sell gpus to that company. There have already been quite a few criminal cases around the export ban.
seanmcdirmid 9 hours ago [-]
That would be unenforceable though knowing American arrogance they would definitely try. It’s like China trying to stop their rare earths from going to American uses by requiring all companies that import them to track their usage.
LtWorf 11 hours ago [-]
Then china stops selling goods to USA for a couple of weeks until they realise there's more in life than GPUs.
cadamsdotcom 1 days ago [-]
What if the gov tolerated gpu exports on purpose?
Black & grey market devices marked up many times cost buyers heavily, while still suppressing motivation to develop sovereignty.
If nvidia selling to China tolerated for the realpolitik and gets undone by incoming powers to create an appearance of change, is it really the right thing to do? Is it the smart thing to do?
It's an interesting thought experiment, though, we will probably never know.
schainks 1 days ago [-]
> What if the gov tolerated gpu exports on purpose?
Given the track record of this administration, they were probably paid to tolerate said exports.
Why on earth would that be suppressing motivation to develop sovereignty? The whole thing makes developing sovereignty an existential priority. When one critical part of your AI technology stack is subject to whims of your adversary, it's not really fully your AI technology stack. Today they turn the tap partially on, but only unofficially, tomorrow they'll shut it off. Time to buy your own tap.
cowlevel 15 hours ago [-]
I don't know the exact reason, but all empirical evidence shows that it does. E.g. the whole of Europe still using Microsoft because they can, even though bits and pieces get cut off (like the ICC). If Microsoft cut off the whole of Europe, then Microsoft wouldn't have power over them any more.
kspacewalk2 9 hours ago [-]
Microsoft's days in many EU countries' government departments are numbered regardless of what Microsoft does. The understanding is there: a critical dependence on a product coming from a now-unreliable ally is a problem. It will, however, take many years for this to actually happen, because it's much easier said than done.
cowlevel 7 hours ago [-]
That's what they say. What they actually do betrays otherwise.
stale2002 21 hours ago [-]
It would suppress it because investing in this stuff is a very large short/mid term cost for long term gain.
This, providing access to the gpus but at a large markup plays into that dynamic.
Entities being myopic shouldn't be surprising.
kspacewalk2 9 hours ago [-]
Does the Chinese government strike you as an organization that's strapped for cash or infected by short-term thinking?
stale2002 7 hours ago [-]
Yeah actually, they have engaged in tons of short term thinking to their detriment.
Look at their demographics issue, as well as things like the real estate bubble.
Their economy is below what it was supposed to be by a real margin.
humanlity 24 hours ago [-]
Post-Trained To Reinforce Western Values
layoric 1 days ago [-]
Yup, I think the same applies to traffic of arms regulations.. Private firms sell to countries they are 'allowed' to, passing it between organisations in a chain until they get to the original customer they couldn't sell to directly.. Cuts are taken along the way, and people making money look the other way..
epistasis 1 days ago [-]
When Xi visited, Trump offered to sell him US made weapons.
What's the chance that these GPUs got to China without Trump getting a cut somehow?
Pretty much zero. Trump sat on knowledge of the affair for an entire year. That's extremely guilty behavior, he was either extorting Huang or covering g for him.
The whole thing is the most corrupt thing we have ever seen at the presidential level.
esseph 1 days ago [-]
> The whole thing is the most corrupt thing we have ever seen at the presidential level.
Bigger than Watergate by far
whynotmaybe 1 days ago [-]
... So far
stratos123 9 hours ago [-]
Weird detail: some of these results are based on initial patterns in weights surviving through training. But isn't muon, which is used for many of these models, well-known to be able to consistently erase patterns in weights, unlike previous optimizers [1]?
This doesn't really call the overall results of the report in the question (the strongest finding is the perfectly matching weights for dead-weight rows which weren't touched by an optimizer at all), but it does suggest that the explanation in the twitter thread[2] of how this works is way too simplistic. Are the patterns in question somehow "special" and not erasable even with muon?
I have run some large open models on neoclouds and the KYC has been minimal, even if the USG stops all GPU exports it will be extremely difficult to crack down on Chinese renting GPUs on Western servers.
There will be a black market of renting IDs to KYC and gain access to the chips, and then the hosts will monitor their servers for what is being trained, it will be a cat and mouse game.
We are headed for an increasingly unfree computation landscape and the politicians are going to make demands that will be hard to deliver.
anon291 21 hours ago [-]
Matrix multiplication is a human right
octoberfranklin 20 hours ago [-]
Keep your laws off my weights.
saidnooneever 1 days ago [-]
i dont know what is expected of export controls... they are never effective. they also find western chips in iranian drones etc. there is a huge gray area of resellers who resel to resellers etc. its pretty much uncontrollable....
0cf8612b2e1e 1 days ago [-]
During the Cold War, America used various shell companies to get valuable titanium out of Russia for building spy planes. If someone is willing to pay, smugglers will find a way.
olalonde 18 hours ago [-]
A lot safer than smuggling since re-exporters generally don't get arrested unless they travel to the US.
LarsDu88 1 days ago [-]
Simply banning things causes the price to go up. That alone is good enough to motivate the policy decision.
AnthonyMouse 22 hours ago [-]
> Simply banning things causes the price to go up. That alone is good enough to motivate the policy decision.
No twice.
First, banning them doesn't necessarily even make the price go up. If all you have to do to get the product is put someone else's name on the form, that raises the price by a number that rounds to zero. It probably costs your own companies more in bureaucracy and paperwork to try to enforce the pointless rules than it costs the other one to put a different name on the form.
Second, suppose it actually worked somehow. What happens then? The other country pays the higher price temporarily while throwing out some big government subsidies to develop a domestic version of the technology. Soon your own companies have a new foreign competitor in the world market which is receiving large government subsidies.
Export controls are a failed policy.
overfeed 18 hours ago [-]
> That alone is good enough to motivate the policy decision.
This may work when its against an smaller. When deployed agianst a (near) peer - there usually isn't a price to high for a field of national security interest. China could get all the GPUs it wants / the US could get all the rare-earth magnets it wants over the objections of the other country.
LarsDu88 7 hours ago [-]
It can get all the GPUs it wants... at a higher price.
overfeed 5 hours ago [-]
> It can get all the GPUs it wants... at a higher price.
That's implicit in "there's no price too high". However higher the price is, the difference is still a rounding error when your budget is trillions of dollars.
I present this short-form documentary — or animated infographic, if you will — on how much governments are prudent in saving money in areas considered of vital national interests, denominated in GPUs instead of dollars: https://www.youtube.com/shorts/F6jMsMBVB34
wmchen 1 days ago [-]
Export controls inherently raise the cost of doing business with parties you don't like, the other party has to either invest more resources acquiring them, or make the decision to forego acquiring them. The grey market exists precisely because there's an opportunity to exploit this for profit. At the end of the day, when these effects of these export controls are discussed, they usually seem to revolve around export controls setting a specific program or effort back by a span of months, not in stopping a specific commodity from accessing another market whatsoever (excepting some hyper-specific components).
krona 1 days ago [-]
> its pretty much uncontrollable
As Hezbollah discovered in 2024, covert supply-chain interdiction is very possible and I'd be surprised if the CIA didn't know what was going on, at least to some extent.
andy_ppp 1 days ago [-]
Yes or indeed supplying cards to the Chinese market with extras...
HDThoreaun 18 hours ago [-]
Would be nuts but unsurprising if the cia is selling subtlety broken gpus to china
treebeard901 17 hours ago [-]
If you could rent compute to what is essentially a competitor and monitor everything about their code and models in the process, since you're still in control of the hardware, why would a company or in this case a country not take that competitive advantage?
yumraj 23 hours ago [-]
Not doubting that they have access to these.
But, could the fingerprints be due to distillation from Blackwell trained models rather than training?
taylorfinley 16 hours ago [-]
Weights start as random values generated from a seed. Pytorch uses however many cores are available on the architecture to write out the initial values in parallel, so each seed + architecture pair has a particular signature. If I understand Will's technique correctly he's basically made a rainbow table of initial weight states. Training alters the initial weights, but if you find one seed + architecture pair that vastly more-closely matches the trained weights, you can infer it must have been the source of the initial weights.
Distillation might shape the weights similarly in some way to the source model, but that would come from the signal, not the noise. I think it would be very surprising if the initial untrained weights were encoded with any fidelity into the outputs such that training another randomly-initialized set of weights (likely of completely different size and shape) on them could leave a signature of the source model's initial state, does that make sense?
Tostino 23 hours ago [-]
Not seeing how that could happen since all the "distillation" attacks i've seen are just token level, not logprob level, and definitely not vector level. So, I don't see how that could be the case.
sam0x17 24 hours ago [-]
Just further proof that export controls rarely restrict the ones they are designed to restrict
cowlevel 15 hours ago [-]
The purpose of a system is what it does, not what it doesn't do.
sam0x17 9 hours ago [-]
So the purpose of MS365 was to go down for 12+ hours?
JohnsonZou 22 hours ago [-]
Awesome job, most interesting analysis I've read these days.
mark_l_watson 12 hours ago [-]
This is all so stupid.
US businesses need two things:
1. access to the very best frontier models for hard problems
2. good enough open models from China, the US (I like poolside.ai, but lots of good choices), and even other countries like France that make smaller investments in Digital Sovereignty.
If the ‘super rich fucks’ in the US who have been making risky AI bets succeed in convincing the US government to curtail the use of small efficient models for easier problems, that will just hurt our economy.
netvarun 1 days ago [-]
@dang curious, is this post getting heavily downvoted or flagged?
I get that the geopolitical angle is sensitive, especially with the backdrop of today's big news - but I thought this was a really well-researched technical deep dive, right up HN's alley. (TIL: you could fingerprint gpu hardware just from an llm weights!)
liuliu 1 days ago [-]
I need to double check PyTorch code, I am not sure how robust this fingerprinting is without PyTorch "bug". Parallel philox based PRNG is a well-researched area and I am surprised (even if it is not secure-PRNG) it can fingerprinting like this. But obviously there are more people who are more knowledgeable on this can give their perspectives here.
Alien1Being 23 hours ago [-]
Not too many years before the headline reads
"Western AI models using GPUs export controlled by China"
The decline of the USA and the West has been precipitous.
lynguist 11 hours ago [-]
Ah, that means the recent huge open-source AI advances, that we all benefit from, directly and indirectly, come from Jensen Huang's China boycott circumvention.
Which means, in future, we maybe cannot expect such large capability increases.
nullc 1 days ago [-]
Cute trick but it's not exactly a secret, e.g. deepseek recently published their own inference engine -- and it only has kernels for SM100 (blackwell) and Huawei Ascend 950, and all its published performance results using blackwell.
No one found it remarkable a couple weeks ago, ... but now the fact that DS is using blackwell is apparently some super secret requiring clever RNG init stats???
Cameri 1 days ago [-]
So export controls are more like export friction
1 days ago [-]
brcmthrowaway 24 hours ago [-]
Are (ex-)OpenAI/Anthropic employees leading the charge here to protect their portfolios?
glennfiedler 23 hours ago [-]
Ladies and gentlemen, we are now officially in Huang-gate.
sandos 14 hours ago [-]
My BS-meter just went through the roof:
"Specially configured AMD hardware, which is also restricted from export to China, is the only alternative we have identified; we have excluded it for DeepSeek-V4-Flash"
This sounds as if they are using something VERY fuzzy to "match" things, and if they are ALSO matching against completely different hardware which this makes it sounds like.... then I dont know if this means anything.
adrian_b 11 hours ago [-]
They match the pseudo-random generator from the NVIDIA library cuRAND.
AMD has a library that emulates it, for compatibility.
Their methodology is interesting, but it only proves access to a single Blackwell GPU, which would have been sufficient for initializing an LLM with random values, before the start of its training.
There is no evidence whatsoever about which GPUs were used for training, except the supposition that if they had access to 1 Blackwell, they might have had access to more of them. Such a supposition may be justified about the number of cockroaches in a kitchen, but less so about the components of a training cluster, which might be heterogeneous.
The LLM could have been initialized on the workstation of a developer, before being transferred to a server cluster, for training.
flakiness 23 hours ago [-]
Such a great domain name by the way.
elonfboy 24 hours ago [-]
WAIT! Are you saying the giant plagiarism machine was built with…stolen IP?!?!
octoberfranklin 20 hours ago [-]
Title is deceptive; at most they've shown it carries the fingerprint of "a 148-CU GPU".
entrope 18 hours ago [-]
No, they distinguish AMD vs Nvidia based on rounding or some similar difference farther down in the stack, finding that the values that would differ based on chip vendor are all consistent with an Nvidia platform. Search for "slightly differently".
rubberstamp 16 hours ago [-]
Land of the free*.
*conditions apply
htrp 18 hours ago [-]
clickbait written as research....
throwa356262 17 hours ago [-]
I’m best known for my work at OpenAI, where I helped create Sora and Sora 2
He probably owns tons of stocks that will increase in value when Chinese models are banned.
yogthos 21 hours ago [-]
oh no, anyways
piokoch 15 hours ago [-]
US frontier AI models carry the fingerprint of data acquired from illegal sources like Anna Archive, libgenesis, etc. and, (un?)surprisingly, nobody cares.
Only Anthropic was stupid enough to let be caught red-handed but get away with some ridiculously low penalty.
ftchd 23 hours ago [-]
Well, d'uh...
apishare_api 1 days ago [-]
[flagged]
exabrial 1 days ago [-]
I guarantee you this isn't an "accident". Whatever is being sent over there is loaded with its own special spyware lol
fibonacci112358 1 days ago [-]
[flagged]
chews 1 days ago [-]
We need deep investigations at the CFTC and SEC level on who took short positions and put options on NVDA. My bet is there are Singaporean propshops backed by the likes of Goldman and others who are rightsizing this market.
SpaceManNabs 1 days ago [-]
didn't the culper report come out in march or something like that?
Rendered at 00:24:30 GMT+0000 (Coordinated Universal Time) with Vercel.
“In facilities across the region, Megaspeed rents Nvidia chips to Chinese tech giants such as Alibaba, an arrangement that does not violate current US export control regulations.” See [1].
[1] https://www.atlanticcouncil.org/dispatches/beyond-the-spy-sc...
To be clear: I am not saying any of this is fair (or legal); I am only explaining how it works.
The same way they "jail" UN-appointed judges that go against their masters' interests: de-platform them from all US-controlled payment and software platforms.
if you are referring to people at the ICC, they are cut off from US services, but that is a long way from being in jail.
> He said: “We are going 30 years back in time. It’s like a time machine returning us to a pre-digital world,” he told France TV.
https://vinnews.com/2026/03/23/icc-judge-who-issued-netanyah...
There's no gotcha here, companies the world over could just ignore any sanctions they wanted, and some try to, but they do so knowing they themselves are (potentially) breaking local and international laws.
You might have well asked "what if a company helps you by importing contraband instead of you doing it directly"
Given the USA has completely torched its reputation amongst allies in the space of a year, I would expect this to happen more often.
Does anyone know why does the US allows Canada to trade with Cuba? That one always stood out to me as a bit odd. My guess: Canada is the largest trade partner with the US, so the US probably has more to lose by sanctioning Canada.
They do try workarounds, and when those are caught there are consequences
You can sit back down now.
Edit: Is there going to be more "laws can be broken so let's not bother" arguments, because that's lame and boring
Largely because other countries are willing to go along with it, and to the extent they are willing to do so.
Even NATO countries import Russian gas. The UK rolled back sanctions on Russia to import diesel earlier this year. Most of the rest of the world did not impose effective sanctions and lots of people trade with them (including huge economies like China).
The rest of the world is to dependent on China to want to get involved. The US might succeed in the current administrations attempt to split the world into new cold war spheres of influence (because people are scared of China too) but its not going to be easy, it will be painful, and its not going to get many countries to completely agree to what the US wants.
Lol. Yeh we can all see how well the sanctions worked on Russia and Iran. Thanks for proving my point!
I was wondering why you wrote this part in French instead of English. Turns out it is a phrase that is sometimes loaned into English. It comes from the (French) writings of Voltaire.
https://wordhistories.net/2016/10/22/pour-encourager-les-aut...
https://archive.ph/BiSrT
"A key player in getting Megaspeed off the ground was Alibaba, the Chinese e-commerce leader that has also been an aggressive player in AI, said people familiar with the negotiations at the time. The U.S. had begun imposing export controls on advanced chips in late 2022, preventing Nvidia from selling them directly to China over concerns they could boost China’s military and surveillance capabilities.
"Alibaba was looking for a partner that could help it access computing power from data centers stuffed with Nvidia chips located outside China, the people said."
I have no idea whether this is legit or not (there's no link to any code), I'm mentioning this because all the comments here are assuming truth without verifying the contents.
The supposed “China Hawk” you mention, on google searching, seems to be David Perdue.
Is there a chance you are confusing similar yet different last names as the same person?
There isn't even an allegation that the Chinese companies lied about this. They simply didn't disclose which chips were used. In all cases where they did disclose the training chips, this research found the disclosures to be accurate.
The method just strongly suggests that the initial values of the weights, before starting training, were generated with the NVIDIA cuRAND library on a Blackwell GPU (or on an AMD GPU, where the NVIDIA library is emulated for compatibility).
The training might have been done with thousands of GPUs of other types. It is plausible that the GPUs used for training or at least a part of them may have been of the same type with the one used for initializing the model, but it is not proven in any way.
A scenario that matches what is reported in the article, but in which there is only a single Blackwell GPU involved, is if the LLM was intialized on the workstation of some lead developer, who had the privilege to use the best GPU they had, and then it was transferred to a server cluster for training, where other kinds of GPUs were used.
Now that this has been published, it would be very simple in the future for any Chinese AI company to avoid this "fingerprinting", by just using a different PRNG, instead of cuRAND.
There were cases being considered as recently as yesterday. https://www.icc-cpi.int/case-records
Yes it has nukes. However china also has nukes. That poses some legal limitation.
One bigger aspect of this is that it was all saved from being leaked until right after the election.
Somebody was playing partisan politics with national security.
One key question: did Trump recently sell NVDA? How much and when? He's known of this for more than a year.
I think the frontier models are definitely in a healthy dynamic where it seems the near-but-not-quite performance of open-weight models are what is pushing the most innovation and price competition from OpenAI, Anthropic and co
If GPUs are going to be priced out of consumer reach anyway, I would rather see the GPUs used to create open weight models than just to make these AI companies richer.
Their government is not really the type to violently take over other countries and impose their laws. Taiwan is a special case because they see it as a rebel state on their own land. The rest of us are probably better off with China being the top superpower than Trump's America.
From my (very cursory, naive, layman's) glance at the summary, they're identifying and tracing some characteristics that specific hardware imparts into a model's initialized weights. But then concluding that this indicates the models were also "trained on" that same hardware.
But is that a valid assumption? Is it possible that a model was initialized on one system configuration, while the bulk of the training was performed on another?
In fact, it would seem reasonable for the "okay, I'm setting up this new model on my workstation, prepping it for training next month" pipeline step to be far removed from the "okay, now deploy the [initialized, but not yet trained] model across our giant fleet of beefy training machines".
Yeah makes sense. They are in transition. This article doesn't really tell us anything new (or interesting). There's nothing wrong w Chinese labs using GPUs that are already in the country. An export ban just means reduced import of new GPUs
The 2nd option is perfectly legal and allowed by US government.
What is surprising, I guess, is that the US allows GPUs to be exported at all to most countries. Since...as long as they are exported to a country that is friendly to china (which is basically most countries on the planet), China has access to GPUs on which to train with, making GPU purchasing restrictions mainly symbolic.
Black & grey market devices marked up many times cost buyers heavily, while still suppressing motivation to develop sovereignty.
If nvidia selling to China tolerated for the realpolitik and gets undone by incoming powers to create an appearance of change, is it really the right thing to do? Is it the smart thing to do?
It's an interesting thought experiment, though, we will probably never know.
Given the track record of this administration, they were probably paid to tolerate said exports.
This, providing access to the gpus but at a large markup plays into that dynamic.
Entities being myopic shouldn't be surprising.
Look at their demographics issue, as well as things like the real estate bubble.
Their economy is below what it was supposed to be by a real margin.
What's the chance that these GPUs got to China without Trump getting a cut somehow?
Pretty much zero. Trump sat on knowledge of the affair for an entire year. That's extremely guilty behavior, he was either extorting Huang or covering g for him.
The whole thing is the most corrupt thing we have ever seen at the presidential level.
Bigger than Watergate by far
This doesn't really call the overall results of the report in the question (the strongest finding is the perfectly matching weights for dead-weight rows which weren't touched by an optimizer at all), but it does suggest that the explanation in the twitter thread[2] of how this works is way too simplistic. Are the patterns in question somehow "special" and not erasable even with muon?
[1] https://docs.modula.systems/examples/weight-erasure/
[2] https://twitter.com/willdepue/status/2120992457630326841
There will be a black market of renting IDs to KYC and gain access to the chips, and then the hosts will monitor their servers for what is being trained, it will be a cat and mouse game.
We are headed for an increasingly unfree computation landscape and the politicians are going to make demands that will be hard to deliver.
No twice.
First, banning them doesn't necessarily even make the price go up. If all you have to do to get the product is put someone else's name on the form, that raises the price by a number that rounds to zero. It probably costs your own companies more in bureaucracy and paperwork to try to enforce the pointless rules than it costs the other one to put a different name on the form.
Second, suppose it actually worked somehow. What happens then? The other country pays the higher price temporarily while throwing out some big government subsidies to develop a domestic version of the technology. Soon your own companies have a new foreign competitor in the world market which is receiving large government subsidies.
Export controls are a failed policy.
This may work when its against an smaller. When deployed agianst a (near) peer - there usually isn't a price to high for a field of national security interest. China could get all the GPUs it wants / the US could get all the rare-earth magnets it wants over the objections of the other country.
That's implicit in "there's no price too high". However higher the price is, the difference is still a rounding error when your budget is trillions of dollars.
I present this short-form documentary — or animated infographic, if you will — on how much governments are prudent in saving money in areas considered of vital national interests, denominated in GPUs instead of dollars: https://www.youtube.com/shorts/F6jMsMBVB34
As Hezbollah discovered in 2024, covert supply-chain interdiction is very possible and I'd be surprised if the CIA didn't know what was going on, at least to some extent.
But, could the fingerprints be due to distillation from Blackwell trained models rather than training?
Distillation might shape the weights similarly in some way to the source model, but that would come from the signal, not the noise. I think it would be very surprising if the initial untrained weights were encoded with any fidelity into the outputs such that training another randomly-initialized set of weights (likely of completely different size and shape) on them could leave a signature of the source model's initial state, does that make sense?
US businesses need two things:
1. access to the very best frontier models for hard problems
2. good enough open models from China, the US (I like poolside.ai, but lots of good choices), and even other countries like France that make smaller investments in Digital Sovereignty.
If the ‘super rich fucks’ in the US who have been making risky AI bets succeed in convincing the US government to curtail the use of small efficient models for easier problems, that will just hurt our economy.
I get that the geopolitical angle is sensitive, especially with the backdrop of today's big news - but I thought this was a really well-researched technical deep dive, right up HN's alley. (TIL: you could fingerprint gpu hardware just from an llm weights!)
"Western AI models using GPUs export controlled by China"
The decline of the USA and the West has been precipitous.
Which means, in future, we maybe cannot expect such large capability increases.
And these were completely openly disclosed as headline properties of the software: https://github.com/deepseek-ai/DeepInfer/tree/main/#performa...
No one found it remarkable a couple weeks ago, ... but now the fact that DS is using blackwell is apparently some super secret requiring clever RNG init stats???
"Specially configured AMD hardware, which is also restricted from export to China, is the only alternative we have identified; we have excluded it for DeepSeek-V4-Flash"
This sounds as if they are using something VERY fuzzy to "match" things, and if they are ALSO matching against completely different hardware which this makes it sounds like.... then I dont know if this means anything.
AMD has a library that emulates it, for compatibility.
Their methodology is interesting, but it only proves access to a single Blackwell GPU, which would have been sufficient for initializing an LLM with random values, before the start of its training.
There is no evidence whatsoever about which GPUs were used for training, except the supposition that if they had access to 1 Blackwell, they might have had access to more of them. Such a supposition may be justified about the number of cockroaches in a kitchen, but less so about the components of a training cluster, which might be heterogeneous.
The LLM could have been initialized on the workstation of a developer, before being transferred to a server cluster, for training.
*conditions apply
Only Anthropic was stupid enough to let be caught red-handed but get away with some ridiculously low penalty.