It's true that there isn't currently a single Chinese model family that matches the full OLMo/Nemotron end-to-end 'cookbook' one could rightly call "open-source" (i.e. a full manual on how to reproduce your own foundation model by including the models, data, checkpoints, evals, and code for base/thinking/instruct/RLVR)...

But! the Chinese ecosystem does have most of those components, just that they're split across separate projects: MAP-Neo or YuLan-Mini for transparent pretraining, plus DAPO/verl or Open-Reasoner-Zero for transparent reasoning RL.

My bet is it's just a matter of time until there is a Chinese equivalent to OLMo. It won't just be China either - from the public sector across the globe there are efforts ranging from Apertus in Switzerland to LLM-jp-4 in Japan, and more. They're not OLMo-level but I'm sure by 2027 you'll see fully open-source 'cookbooks' help an enterprising individual reproduce frontier models of at least a 2023-2024 vintage.

Equating an open weight model to a closed source binary is ludicrous. An open weight model can be fine tuned, for example. How easy is it to modify a closed source binary for your purposes?

Open Source AI is challenging. Even tiny models require tremendous resources to produce, so all the things we know about "open source" for software, like a random person in Nebraska can produce a critical piece of the world's infrastructure in their spare time, don't apply.

Actually open source AI is more like scientific research. It needs public funding and reputable institutions as stewards of that funding. I'm hopeful Allen AI is able to do the things, but I'm not sure the US is moving in the direction it needs to be to get to the point where AI in the public interest is something we invest in.

But, also Photoshop is not at all comparable to open weight models. I can't legally give you a copy of Photoshop, but I can give you a copy of GLM 5.2. There are free proprietary applications out there that fit the bill, Photoshop isn't one of them.

This is a facile and pointless discussion. The hard work in building source code is paying millions of dollars to software engineers to write it. "Open source" allows others to benefit from this expenditure. The hard work in building an ai model is spending millions of dollars on training. "Open source" models allow others to benefit from this expenditure. Regardless of whether you had access to the complete training data and architecture of a model, you would never retrain from scratch in order to make a modification to the model. You would fine-tune in whatever behaviour you wanted. Just like open source software is when you are allowed to modify it, open source models are when you are allowed to fine tune like this.

Especially since some of the arguments in the article seem to hinge on "just fix the training!", yeah, I think this is a completely fair call-out.

Open weights can sometimes get additional training, but you can't remove existing training, so there kinda isn't a fair claim to "just train it to be [nationality]". That would need "real" open source so you can train a realistically-equivalent model from the ground up.

The two aren't comparable.

You can't reconstruct the same LLM weights even with the same training data.

You can fine tune an open weights model.

Much of the value in modern LLMs is in the RL envrionments, not the pre-training data. That is a very different piece of software to an LLM, and the LLM weights are useful without it.

And at the risk of being a downer, I'm not sure Open Source LLMs is actually viable.

Training a model requires two or three orders of magnitude more investment than the typical OSS/Creative Common contributor can reasonably afford.

It reminds me a bit of Open Source hardware (CPU, GPU).

> This is not "open source" AI.

The OP actually acknowledges this in the footnotes. Perhaps we don't need to re-litigate this on every post?

Anyone got a starters guide for setting up an open source LLM? I'm pretty new to AI, but technical. Im looking to setup my own open source LLM on my workstation to help me code if its possible. Been using Claude and I tried opencode recently.

I sort of agree with your analogy but in another sense, think of Firefox.

Could I have made it myself? No!

Am I grateful that I can have the output of Mozilla’s internal efforts — the browser source code — so that I can at least tweak it and rebuild it myself, and even ship the results myself? Yes!

A open weight model is both an incomprehensible binary blob like Photoshop, but also patchable and adjustable piece of source code, like Firefox.

> This is not "open source" AI.

I both know what you mean and I agree with you, but attempts like this to maintain some sort of linguistic purity of meaning never work once the discussion escapes small communities into the larger mainstream.

So you should probably get used to the idea of people saying "open source LLM" when they mean "open weights LLM".

>>There is nothing "open source" about the Chinese models in question.

hard disagree here. openness can have a scale and on that scale Open AI is less open than zai. olmo will be very very open on that same scale.

This is a great distinction that I don’t think is getting talked about enough. “Open source” is probably the wrong term to use. Open “weights”, sure.

If your only concern is how good is it at coding, then I don’t have an issue with using the Chinese models. Especially if you want to run it locally, they are kind of the only choice.

For any other use than coding, it’s going to have to be a hard pass from me.

Chinese models are open-source for purposes of inference: you know exactly what the model architecture is and the code to run it is released under an open source license. The “binary blob” is a useful artifact that has many techniques available to use in the creation of future binary blobs. And it is fixed and reliable.

Contrast this with crap like Claude, where you have no idea how many/what size/shape models you are interacting with. They can silently inject outputs from other systems, apply steering vectors to sabotage a particular user’s work, etc.

I agree that 100% free-range organic open source models are lovely, but they also are unlikely ever to attain parity with the frontier because the training data sources are open and expose them to legal liability that closed model providers can thwart.

When talking about "open source" models no one is talking about the legal framework and textual doctrine of open source. They simply want to know: "can I run this on hardware I own or no?" That's it. It's simply about control. If you don't think there's a difference between models you can download and run on hardware you own, regardless of how big it is, vs. models you can only talk to through an API, you are seeing the forest for the trees.

You are also indirectly helping the closed SOTA model toll-keepers by taking the focus away from their mass copyright theft and onto some pointless textbook definition.

It's a matter of degree. If open source means "everything you need to reproduce this" then maybe an open source application should include a detailed architecture document and course in the c language. Oppositely, you can certainly use models to reproduce model - the Chinese models themselves were supposedly created by interactions with the American models so you can use them/

It’s a good analogy. One difference is that it would be much more difficult to “fine tune” Photoshop, so there’s a sense in which free weights are more useful than a free binary.

The more I see this argument, the sillier it becomes to me. The point of "open source" in software is something like "the ability to deconstruct the thing line by line for purposes of predicting what it will do."

But you can't fully do that for generative AI anyway? What's the point?

Now, if you wanted something more robust, then you probably should have supported Stallman's ideas and insisted on Free Software in the first place.

[dead]

If AI were to become a super weapon why should I trust a private company to own it? If the super weapon is publicly available to download then why should American citizens be banned from doing so?

The OpenAI/Huggingface scenario proves the necessity of open models.

Huggingface, in their postmortem, explained how they quickly realized that the threat was an advanced AI and proceeded to use frontier models to analyze the suspect telemetry, but were unable to due to frontier model safeguards. So they fired up GLM instead, which had no such safeguards and proceeded to analyze their telemetry just fine and get to root cause. This proves without a doubt that companies need models with minimal safeguards in order to properly assess threats. If open weight AI doesn't exist to check out-of-control frontier AI (which was exactly the scenario that Huggingface experienced), then it's game-over.

> Some people would say it is reaching some sort of singularity.

Those people never took Systems Thinking, or are failing to apply it to AI as a non-closed system.

I agree with your overall argument. But I think there's a challenge here - if the "ultimate AI" is now being built in the West, it is being built with private investment from many sources. I think even China depends on investors expecting payoffs. Those sources expect not a single Singularity but profits since these can be divided. Moreover, those investors expect a streams of profits even before the Singularity, so the companies need to sell their models, if they lock this iteration, they won't have the money for the next iteration.

Quite a balancing act.

> there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon.

None of the AI labs have been shown to have any sort of significant research lead over any of the others.

So if progress does eventually lead to a singularity event and an ASI super-weapon, there's no reason to believe that once the first lab hits that threshold, every other lab won't also hit that threshold independently within weeks/months of each other.

...unless the first group to make it there literally intends to murder everyone else working in the space as soon as they hit the (otherwise not-so-singular) singularity.

And if that's the plan, well that's not very nice.

>it's not hard to see

It's also not hard to see that these people need professional help because they're willing to throw literal lives into the meat grinder for just a whif of enlightenment and the smallest chance at deification.

If it's a superweapon, then it's unconstitutional to ban it. Huzzah for the 2nd Ammendment.

I tried to address safety, albeit briefly, in the sections on backdoors and propaganda. What would you have liked to see?

I didn't give it much treatment because my frame of reference is about open vs closed AI, and I don't see a lot of daylight in safety concerns between the two.

> The reason why …

If someone read the OP article and came away presuming there are no legitimate other reasons why reasonable people have safety concerns [1] it seems to show the article did the opposite of informing people about such concerns.

[1] https://news.ycombinator.com/item?id=49029303

Conversely, “we should stop X” should come with some idea of how to feasibly do so.

In any case, I’d summarize my position as “you cannot stop open source AI, and attempts at it will inevitably empower bad actors relative to good ones.”

This, right here, is the bottom line. Money gets what Money wants.

will there ever be a way to fully audit american models? hell, at least they're releasing the weights for these so they actually could be audited!

You have the wieghts for the chinese models. Go ahead and run whatever tests you want.

The american models on the other hand are too powerful for simpletons like us to be allowed to use, but we can be assured its all in our best interest, citizen

Audit them for what, exactly?

Backdoor hardware is straightforward -- its letting people in that the customer doesn't authorize.

Open weights means you aren't tied to any harnessing. So the security domain to be really concerned about is limited to weights only, and I welcome pushback on that.

Are we thinking adversarial injection? Lying about history / propaganda infusion? What is the angle that makes a non-sovereign model dangerous in a way a sovereign model isn't?

> What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?

You mean like ChatGPT hacking Hugging Face? Obviously nothing can stop the closed weights providers from doing "genius-level scams" and in addition you won't know how they did it and what models were used.

In short, only a good guy with open weights can stop the bad guys with closed weights, be them fine-tuned or pre-trained.

What's stopping someone from doing that right now without the LLM, just slightly slower?

Whats to stop me from going down to the local gas station, filling up a few thousand liters of gas in a rented moving truck, and driving into my nearest hospital?

Most people aren't terrorists. You can't just argue stuff is dangerous because it can be one piece of a sophisticated plot.

How can we close pandora's box, now that its open?

Safety is not restricting access to only people favored by the government. The greed of AI executives opened pandora's box. Now they are desperately trying to find ways to reap the benefits with none of the consequences.

I don't believe trusting big brother and big tech is the solution either. Besides, even if it's open weight, it still runs on someone else's hardware.

Unless you have the money to run them locally, and if you do, you could do a lot worse than scams. Ask any lobbyist.

Amazing how we quickly the crypto wars have been forgotten and the lessons not learned.

It's already an established industry with human resources. If you read news about how these scam factories operate in Southeast Asia, their cost to use real humans is even cheaper than running a 2.8T model locally for scams. You don't need genius-level intelligence to run scams. Maybe we tend to overestimate how smart people are.

Nothing, which is already happening with weaker models which are already released.

their reputation, future and business and the fact that people can go somewhere else, just like anywhere else in the open source world. What stops RedHat from putting genius level scams into their operating systems? The fact they're done after they'd pull of their first scam.

Obviously if you're using something anonymously published to the internet you can get scammed the same way you can get scammed on Craigslist, so just don't do that

There are a number of distinctions: https://news.ycombinator.com/item?id=49029303

It could be stopped, if there were a social movement large enough to make AI a taboo.

The difference is that people building atomic weapons knew what they were building. OpenAI/Anthropic have no idea how emergent intelligence works. They're just running in the dark towards the cliff. Leaving it in their hands is the same risk as leaving it to any random guy, because random guy and the labs have the same understanding of what they're building. I'm tired of us treating labs like they're somehow uniquely good at AI. They are not. They simply have enough capital to train. If training was affordable they would have no edge whatsoever.

This argumentation is going to be joked about in the future. Like downloading a car.

Would you open source a nuclear bomb???

It's also not trivial to tell if an open source project has had malicious code added. It might be a bit easier than to tell if there's malicious code than if the weights in a model were poisoned, but for 99% of users, you're going on pure trust in both cases.

And in both cases, even this seems better than relying on a closed source, service only solution where the same issues could be completely undetectable (at least without way more analysis)

> It isn't trivial to tell if the weights have poisoned:

True, but it is easier if you have the weights than if you don't. I guess this was their point?

Exactly. Faced with a choice between open weights that might be poisoned and walled-off weights that I know are poisoned... well, it's an easy choice.

Regarding encryption. Interestingly, US government also insisted on DES and AES being open standards, because using bad encryption did more harm than good.

designs on the time scale I guess. and openai might feel secure in their favor with the admin?

[dead]

"For example, rhetoric relating open source to communism is humiliating to discuss"

All Open Source isn't communism. The GPL license is a form of digital communism. It forces you to open source any additions made to open source code as a way to make things 'equal'.

I'm glad it and Stallman are mostly in the dust heap of history and better licenses, like the MIT, BSD, and Apache have gotten popular.

The irony is that the only open source projects surviving long term are funded by very large corporations.

If you don't care about creating code, none of this should really matter.

My level of trust in Anthropic/Google/OAI/Microsoft/Meta not to do bad things is about zero, so why should I trust them with this?

So? The primary thing stopping people from doing really really bad things is, and always has been, most people not wanting to do bad things.

It's never been difficult. Literally, right this second, you could grab a pen off your desk and pretty easy ram it into the neck of your nearest coworker. Killing's easy. You don't need some hyperintellect to tell you how to do it.

The reason you haven't rammed a pen into your coworker's neck is the same reason why most possible really bad things don't happen: People don't want to do them.

I already know how to kill thousands of people. It's not conceptually difficult: more than a day's effort, sure, but what stops people from doing this kind of thing is not the lack of knowing how. https://xkcd.com/1958/