Not very keen that the whole story of trust is basically centralisation - requiring "big cloud" and essentially promoting lock-in.
Did I get that right? Seems a bit anti-pythonic.
tancop 12 hours ago [-]
I just don't get why the model is absolute trust in a single publisher. They should enforce reproducible builds and set up an attestation system where you can have GitHub or Google build your code and declare that the hash matches what you said.
That way projects that want to be extra safe can set up multiple providers to check the code independently, and third parties can upload their own attestation that PyPI doesn't trust by default but you can add them to your private whitelist. And they wouldn't have to care about where the release is uploaded from.
Crypto projects know that 1 of 1 is the worst possible trust assumption. If you want real confidence you need more than one party.
woodruffw 10 hours ago [-]
Given infinite time and engineering talent, PyPI could do a lot of things. The things we actually do reflect the outer envelope of what we can do.
(Python packaging isn’t anywhere close to reproducible, and it isn’t clear how PyPI would enforce that without pissing off >90% of the ecosystem.)
zbentley 11 hours ago [-]
Reproducible builds only solve small slice of the problem. Build attestation in general solves more, but is useless unless a credible entity is (conceptually) attesting that a package is OK.
The growth of centralized authorities arises from the reality that the vast majority of devs do not and will not independently vet or attest to the quality of libraries they use. That’s not great, but it is what we have and doesn’t look likely to change.
Given that, and given that people still want as much security as they can conveniently get, centralized control (from “we run a security scanner and trust its findings” to “packages can only be downloaded from pypi/debian stable/whatever”) is a natural result.
duskdozer 1 days ago [-]
I think at this point the group of devs+ who are not fully on board with control by and dependence on big tech/VC is a small niche unfortunately.
my-huge-pony 1 days ago [-]
I don't think this is true. From my experience, developers are the people who care about this topic the most. Just look at HN, almost everyone (who speaks out) is against.
akerl_ 1 days ago [-]
I'm very in favor of this work, but don't generally comment because these comment threads are full of folks strawmanning the work being done, and in my experience those folks aren't really interested in productive discussion.
I'd be wary of trying to estimate the split of opinions based on who comments.
rcxdude 24 hours ago [-]
Can you explain the misconceptions? Currently what I see is a) a checkmark box that can only come from a large provider, and b) no good idea of what that checkmark box means in a practical sense, or why I should care or why pypi cared to implement it. What is the threat that this prevents? To me this seems to not particularly improve security but that it does create this pressure towards using a small subset of providers (even if that pressure is pretty light at the moment).
(I've had a quick read through the various links but they mostly seem to emphasize what it's not, which is why I say that it doesn't seem to improve security, and especially I don't really see a motivation for why these other parties need to be involved in the way they are: the only benefit I can imagine is a third-party attestation that some artifact corresponds to some source code, which seems to be partly besides the point and partly solvable through better means)
akerl_ 23 hours ago [-]
I didn't reference misconceptions, for what it's worth. I referred to strawmanning, and if you're curious for some examples:
If you're curious to dig into what the checkmark means, why you should care, and why they implemented it, I'm not going to be able to do a better job than https://docs.pypi.org/attestations/security-model/
rcxdude 12 hours ago [-]
All that page really says is "the signing does what it does", i.e. that it verifies that the package was signed by one of the 'trusted publishers'. It doesn't do anything to explain why I should care if a package came from github or whatever. Especially it doesn't explain why it should be github and not pypi or the original author that does this signing, since really what I care about is whether the author/maintainer of the package has changed, not their publishing platform, and presumably pypi already has at least the means to authenticate them.
lexicality 12 hours ago [-]
FWIW they have at least significantly reduced the size and prominence of the checkmark. The original design had a giant green checkmark front and centre with a bunch of smaller green checkmarks all showing how super special and trustworthy the project was.
the point here being small service providers are _also_ able to get into this system, but I think there's an expectation of success.
The bar for trusted publishers is very high though. I don't think it's high "on purpose" to leave out certain platforms for being too small, but in effect if (for example) your provider doesn't have a security contact point then that's going to rule them out here.
kapilvt 20 hours ago [-]
I’d like to see support for trusted publisher signing and manual upload. Think of it as two step auth for artifact publishing. Pipeline generated artifacts and human upload of those verifiable artifacts. The current reality of trusted publishing, is devoid of trust, except it was triggered by a pipeline that pypi trusted. That pypi is now able to call out such trust is great btw, but as a project that currently does this two step generation, it feels like we are penalized for trying to introduce additional steps out of abundance of caution… separately, what mechanism does a client installer have to verify a pkg install was cryptographically attested to wrt to production?
sans client verification of signatures by default, install is still roulette, and I’m glad package managers have introduced package publication date based cool off options now.
rtpg 18 hours ago [-]
Yeah that's not the worst idea, since in theory signing is about tamper prevention.
(Personally I think that we just shouldn't have automated releases via CI, full stop. There should "always" be a person pressing a button to release things. So I'm in a bit of a big tent with manual upload support. At least some second factor for releases!)
I think generally though the idea is that trusted publisher signing + upload is about simplifying .... something. But given how attacks have been happening in the past 24 months I fear that this might not help as much as we want.
BigTTYGothGF 12 hours ago [-]
> Seems a bit anti-pythonic
Quite the opposite. "There should be one-- and preferably only one --obvious way to do it".
sublimefire 15 hours ago [-]
For example MSFT (not a small player i know) is having its slightly different approach to those signals and they cannot be used here (search scitt). I mean different type signature envelopes, different roots of trust. But in the spirit it is the same. This approach is partially a territorial grab by google (first npm, then pypi, next golang). But importantly I think these security improvements are marginal and you are at the mercy of the dependency developer anyway. Just look at npm, they had some nasty vectors and similar “improvements” were close to useless in those cases.
bbor 1 days ago [-]
It's anti lots of things, but I don't think it's anti-Pythonic. Python is about pragmatism & empathy if it's about anything -- bitcoin and such don't factor into it. See: Astral/OpenAI, Anaconda, most of the PSF board historically being employees of big corporate users of python, NVidia's relation to the ecosystem in general, etc.
As a passionate anarchosyndicalist, I, for one, am thrilled about it. Decentralization is a tool in our toolbox for identifying and dismantling unjust hierarchies, but it is not a panacea, and can be quite damaging when taken to the extreme. See: cryptocurrency speedrunning 200y of bank regulations in 20y.
To put my trust for Python in more concrete terms: they support GitLab. As a passionate GitLab shill, I really couldn't be more impressed! It continues to shock me how infrequently the second most popular forge is acknowledged, much less supported. This shows they're not just, IDK, bowing to Microsoft's demands.
woodruffw 1 days ago [-]
> This shows they're not just, IDK, bowing to Microsoft's demands.
Not only don't we, but I must emphasize that Microsoft did not pay for anything involving the implementation of either Trusted Publishing or attestations on PyPI. I wrote a personal blog post explaining some of my own rationale for why we prioritized GitHub and GitLab for PyPI's initial support targets, and it shouldn't be surprising to anybody who knows me that the reasons are pragmatic, not dogmatic[1].
Is "a public key that I personally upload to my PyPI account, using the credentials you already have that allow me to use that account" on the list of targets that will ever be supported?
If not--i.e., if only recognized "forges" are ever going to be supported--why not?
All absolutely reasonable. At the same time though there are a lot of roads leading to centralisation, and as they all converge it gets just a bit .. problematic.
Absolutely not claiming its not a pragmatic step, but equal-footing alternatives would be useful.
I'm no tin-foil-hat - but when a website account requires government id (like in the uk), and creating software requires signing by the website, you link all the things - is that a good outcome?
woodruffw 1 days ago [-]
This is true, and it’s why I’m hopeful that some of the newer forges can make meaningful default posture improvements that make migration away from centralized services the dispassionate choice to a user who cares about availability and security.
bbor 20 hours ago [-]
Yeah but aren't we talking about PyPI, here? I love some illegal software (shoutout to Anna) but it seems like leaning on various cyber-sensitive registries to make it more available is a very small reward in exchange for a huge, tangible risk.
AnnasArchive in particular is a decent example to keep in mind, since she seems to host her own GitLab instance for opensource development. Obviously that doesn't solve this exact problem b/c self-hosted instances don't work as trusted publishers, but it speaks to what I've clumisly been trying to, here: when it comes to protecting software & speech that the government is trying to ban, making sure those people can host their own stuff and still access the minimum shared infrastructure to allow people to find and use/read it is a lot more practical than trying to build the 99%'s platforms around criminality.
Cause that's what we're talking about, here, to be clear. Criminality. Again I love some criminals and even some crimes, but the word's typical connotations seem like a pretty good reminder of what prioritizing such usecases in the mainstream can cost us.
We know why PyPI ripped out PGP support and has refused any verification mechanisms that other distributions use for decades.
cozzyd 1 days ago [-]
I trust myself way more to with with a pgp key on my computer than figuring out how to use GitHub actions properly...
theteapot 1 days ago [-]
The problem is no one else does. Github workflow is at least public and auditable. Your personal computer is not.
pdonis 1 days ago [-]
So what? Ultimately the trust model is supposed to be there to validate that this latest blob of code is coming from the same source as the last blob of code for the same project, authenticated with the same public key. And that's all it is telling you, and all that it can tell you, whether the attestation is generated during a Github workflow or by a program running on my personal computer that you can't see. It is not telling you that this latest blob of code is safe and free from problems. It's just telling you that it comes from the same source.
akerl_ 1 days ago [-]
The average person using GPG isn’t actually providing me much evidence in that regard, given the number of ways to have technically used a GPG key to sign an artifact that aren’t secure.
pdonis 1 days ago [-]
That's easy to handle: if the signature isn't secure, it doesn't count as an attestation. I have no problem with putting the onus on the person doing the signing to do it right. But at present doing it right on your own is not even an option. Only going through one of the tech giants is.
akerl_ 1 days ago [-]
How is PyPI supposed to tell if the person signing the artifacts is handling their key securely?
tsuraan 7 hours ago [-]
If the PGP key was generated on a yubikey, then the owner of the key can generate and publish an attestation certificate[0][1], and then anybody (including PyPI, I assume) can verify and "gold star" artifacts signed by gpg keys with verified attestations. Pulling a private key off a yubikey isn't super easy, at least not that I know of. Maybe other hardware keys can do similar things?
Seems more straightforward than custom stuff built on OIDC, but I'm not a security guy, so maybe I'm missing something?
One missing bit of context here might be that PyPI used to support GPG signing your packages and barely anybody did it. And that’s not even with trying to push for hardware keys to limit user ability to make a mistake.
You can also look to FIDO2 / passkeys to see that a lot of the folks who have concerns about PyPI trusted publishing also view device attestation as a scourge.
pdonis 1 days ago [-]
They can't. But they can't tell for sure if large tech corporations are doing that either. Data breaches are a thing. Auditing Github workflows doesn't prevent them.
akerl_ 23 hours ago [-]
These aren't really comparable things.
The fact that a large company can mess things up doesn't change the fact that it's way more viable to review how Github or Gitlab or Codeberg handles key material and security than it is to make a comparable local developer flow that avoids common mistakes like just dropping the signing key on disk on whatever servers you want to run your builds from.
pdonis 23 hours ago [-]
> it's way more viable to review how Github or Gitlab or Codeberg handles key material and security
Only if you already trust them.
> dropping the signing key on disk on whatever servers you want to run your builds from
I run my builds on my local machine. The private key I would use to sign them if PyPI supported that is not anywhere in the repo I'm using.
Of course you would have to trust that I'm telling the truth in what I just said. But, as above, you also need to trust what the large tech forges are telling you in order to review how they handle their keys.
cozzyd 1 days ago [-]
Microsoft can't even figure out how to send email reliably. But sure everyone should trust them with signing packages.
UltraSane 21 hours ago [-]
Every private key not created inside a HSM should be assumed to be compromised.
usr1106 15 hours ago [-]
Github workflow public and audible?
The repos workflow source, yes.
But the workflow engine implementation? I don't know how much of it is open source, but I am sure nobody external can audit what Microsoft really runs there. Another company starting with M has just been fined for accessing users' photos. So I would not be 100% surprised if Microsoft changed/amended some user code before executing it or interpreted it in an unpublished way e. g. "for security reasons"
bbor 1 days ago [-]
1. "ClosedAI" is not gonna win people over to your side, IMHO. Apple doesn't plant trees, Microsoft makes big hardware, and Python is not satanic.
2. I'm not entirely sure what the link is supposed to prove -- would appreciate a word of clarification if you find the time. It almost feels like a dogwhistle of some kind, but A) I doubt anyone on HN is anywhere near that problematic, and B) there's no clear pattern in those companies other than "interested in ID tech", and none of the usual dogwhistle targets made the list. Also, y'know, OpenAI isn't on it...?
I guess my fundamental confusion isn't about the list and more about what "we know". Are you implying that they are against pure, unmanaged PGP for this system because... I mean, why? Because they're playing the long game to promote a verification/attestation/identification (?) technology that you think offers less stringent privacy guarantees, and thus to sell more ads in 2036?
If so, doesn't that seem like a kinda roundabout way to raise Display Ads revenue, otherwise known as "that one dying industry"?
asdg164 1 days ago [-]
[flagged]
bbor 1 days ago [-]
Are... are you creating new hackernews accounts for every comment? Why?
Anyway, sorry to confuse/frustrate, I was being honest and tried to explicitly not accuse you of dogwhistling. I was making a best-effort attempt to explain my confusion, rather than just replying with "that doesn't make any sense" or something.
I guess I'll briefly elaborate just in case: "dogwhistle" above refers to a specific kind of antisemetic practice where people point at a list of companies or institutions or even just people and say something like "what do you notice?", whith the message ofc being that they're all related-to/run-by Jewish people. Again, not accusing you of that -- those people don't generally come here. But absent any other logic, I am left with confusion.
I will briefly add that this very response also doesn't help, when you very clearly imply that I am "shooting down a comment" in bad faith for some sinister purpose. It, again, comes off as conspiratorial. Why would I --a sad 30 y/o child with my real name and phone number in my bio-- be shooting this down?
I guess my overall response is that this feels like a conspiracy theory without identifying the conspiracy.
cowlevel 13 hours ago [-]
Creating a new account for every comment is done by spammers to avoid shadowbans. It doesn't work well.
This is great! I want sigstore-based or at least sigstore-style "keyless" short lived signatures to become more and more popular, so that support gets better everywhere. In particular, if having secure OIDC handling in a CI/CD pipeline becomes something that keeps people on GHA or GitLab then I'd hope third-party hosted CI/CD services begin to take that gap seriously and work on meeting those requirements.
I'm bought in on keyless signing, and I want there to be many more providers, so I like to see more adoption which builds more momentum.
rcxdude 1 days ago [-]
>I'd hope third-party hosted CI/CD services begin to take that gap seriously and work on meeting those requirements.
How is a third party service, especially a self-hosted one, expected to meet those requirements? They seem to be lengthy, onerous, and also require a detailed vetting by the pypi team that is unlikely to scale beyond a few large hosting providers.
(What's more, this does relatively little to actually signal trust to someone downloading them. All it does is provide a signature that one of the big providers has built the package from a given source (but nothing about the trustworthiness of that source). This is a problem much better solved by reproducable builds and distributed verification of those builds)
ImJasonH 1 days ago [-]
Tangled and Codeberg both seem to support issuing JWTs during actions runs.
I think OIDC tokens from workflows is basically table stakes at this point, since the alternative is a static secret, which is basically guaranteed to leak eventually.
All that's left is for PyPI to accept these tokens in their set of trusted publishers.
woodruffw 1 days ago [-]
Please see the other comments in this thread (as well as on PyPI's issue tracker) for some important context here. It's not enough to throw JWTs over the wall, PyPI also needs to know that it can trust any given service to operationalize the ugly and risk-prone parts of managing PKI.
(I haven't looked at Tangled at all. If Tangled has a secret management story for its PKI that's stronger than a private key on disk and can demonstrate that a non-trivial number of PyPI users would benefit from federation, then it's a strong candidate.)
ImJasonH 1 days ago [-]
Sorry I didn't intend to insinuate that pypi should necessarily accept any old tokens just because they're there. Just that increasingly OIDC is more and more an option, which is generally a good thing.
PKI definitely is still hard, but if it can be done well by a smallish number of trusted parties to the benefit of their users, this can have huge ecosystem benefits. Let's hope we live in that timeline.
mhils 1 days ago [-]
Very cool to see PyPI put more emphasis towards exposing provenance info. It would be even cooler if trusted publishing would finally support reusable workflows (https://github.com/pypi/warehouse/issues/11096).
pdonis 1 days ago [-]
It would be even nicer if there were a way to have an attestation without publishing with a workflow controlled by one of the tech giants. There's no way for me, someone who has had a PyPI account for many years and has used it for all that time to upload packages, to attach my own public key to my PyPI account and then sign uploaded distributions with my private key so the signature can be verified and count as an attestation.
angry_octet 1 days ago [-]
I think the problem is this type of attestation is hard to weigh as a trust signal, vs the cloud CI version.
Inherently, if someone steals your key, or gains access to a box that can use that key for signing, they are indistinguishable from you. Hand PGP signing also lacks time-bound signing, any ratchet mechanism, CT log, consistent key hierarchy, or standard to bind signer name and project identity.
pdonis 1 days ago [-]
> this type of attestation is hard to weigh as a trust signal, vs the cloud CI version
In the sense that I'm not a large corporation and nobody else is auditing my personal computer, yes, that's true. Note, though, that large corporations are also subject to nefarious actors misusing their infrastructure. What if a rogue employee hacks their internal system? Or Russia or China manages to get a backdoor into it?
In any case, all this doesn't make the method I described impossible. It just means that attestations produced by an individual workflow need to be clearly labeled to distinguish them from attestations produced using a tech forge's workflow, so people will understand what, specifically, is being attested to. Then people could weigh each trust signal as they choose.
> Hand PGP signing
To be clear, I wasn't talking specifically about this method of signing with a personal key. I agree there are lots of issues with PGP as an overall system. I was only talking about the general idea of having some way for individuals to independently generate an attestation without having to go through a tech forge's workflow.
angry_octet 15 hours ago [-]
I think their system is vague, subject to political pressure to revoke attestation, and confusing.
We definitely should have a system that has multiple mechanisms for authorship attestation, of ongoing identity control, code control and binary generation sanctity. The last step can be made repeatable if we can manage reproducible builds.
ghosty141 24 hours ago [-]
Use a yubikey and the problem is solved for example. Also compromised private keys are quite a high bar.
angry_octet 15 hours ago [-]
If you generate the GPG key on the yubikey (every tutorial says not to do this unfortunately). Practically, hardly anyone does it securely with a HSM/yubikey, and they store their key online, where it is collected by info stealers. Even if someone uses a Yubikey, they can prove that they did. The lack of a CT system means signing is secret and not timestamped. We need to do better.
tzlahp 1 days ago [-]
If only there had been PGP signatures. Oh wait ...
duskdozer 1 days ago [-]
Maybe they'll implement PGoogleP, where your private key is safely and securely stored on Google's servers
cowlevel 1 days ago [-]
This is about detecting formerly benign projects that have had their pypi entries hacked, but not their private keys. It does not detect malicious projects.
Kwpolska 1 days ago [-]
And about detecting projects which dared to move their release builds away from GitHub.com/GitLab.com.
eesmith 1 days ago [-]
Am I correct in my understanding that the only trusted publishers are for-profit US companies? And these all use proprietary implementations which cannot easily be replicated elsewhere?
Like, if I want to host on Codeberg, or Sourcehut, what is the process needed for those platforms to become a trusted publisher?
As it happens, I build my releases on my own hardware using a pypa/manylinux release. Can I provide attestation, or am I required to use a third-party service to get that "x Without provenance" removed from my entry?
woodruffw 1 days ago [-]
Support for Codeberg is being discussed here[1]. I think there's broad consensus among PyPI's maintainers and admins that more publisher providers are desired, but we also need them to demonstrate strong key management and rotation controls due to the degree of trust and access that becoming a publisher provider grants them (effectively, they become the PyPI credential broker for the projects that use their service).
(If you pan through that thread, you'll note that one of open issues with Codeberg is that their underlying server wants to store OIDC private keys on disk at the moment. This is okay for small deployments of personal services, but not for something that PyPI needs to federate with and trust.)
So I, as someone who does my builds locally, can never have attested builds?
Will PyPI be presenting warnings or statements of caution which might cause users to be concerned about using my distributions? Or imply that my build practices are somehow inferior to depending on a third-party build system?
guipsp 1 days ago [-]
Yes, that is right - that's the purpose of attesting builds.
Do you see any other alternative?
woodruffw 1 days ago [-]
The current formats (publish and SLSA provenance) are for attesting publishing and build steps, respectively, but there's nothing stopping PyPI from adding a different, new attestation statement for local builds (something that conveys "I did this locally, and I was this principal (email address) when I did it").
pdonis 1 days ago [-]
So are there any plans for PyPI to do this?
woodruffw 9 hours ago [-]
There’s an open issue for it, but nobody is currently working on it. It’s not a mandatory feature or even something that installers current look at; if and when they do I think it’d be a much higher priority subject.
pdonis 7 hours ago [-]
Ok, thanks for the information.
breakwaterlabs 1 days ago [-]
If I understand this correctly the fundamental problem here is what codesigning / pgp solve.
woodruffw 1 days ago [-]
Attestations are code-signing. PyPI once supported PGP signatures on uploads, but as since disabled it because (1) only a tiny minority of users ever signed, and (2) many of the signatures and corresponding keys were either functionally insecure or unusable.
Both of these are latent problems when trying to use PGP in packaging ecosystems; the subset of ecosystems that successfully adopt PGP tend to have very different topologies (namely, small rings of trusted...publishers) rather than an "anyone can publish" architecture like PyPI has.
ashfdgq 10 hours ago [-]
Adoption of PGP would change if PyPI started nagging about PGP signatures instead of some corporate centralized ID scheme.
But that is not done, because python-dev is essentially corporate owned and has always done the bidding of corporations for at least 10 years because their "leaders" want to keep their easy bullshit jobs.
eesmith 8 hours ago [-]
I'll suggest an alternative. There is a strong selection bias for who can become a core python dev. It takes a lot of time, and there is little direct monetary compensation along the way. Most people aren't independently wealthy, so the best way is find a job which pays them to work on Python development.
Those tend to be companies with a lot of money. Those companies tend to have different needs and preferences than a sole proprietor ISV like my company. They tend to hire people who are aligned with their goals. They tend to fund specific PSF projects which are aligned with those goals. Those with strong objections to those goals, or the source of the funding, don't work for those companies, quit, or get fired.
As it is, the entire PyPI infrastructure depends on the corporate largess from the likes of Fastly, Amazon, Microsoft and Google - the latter three of which are described in "From economy of occupation to economy of genocide", Report of the Special Rapporteur on the situation of human rights in the Palestinian territories occupied since 1967, as corporations profiting from the displacement and genocide of Palestinians in Gaza, in breach of international law.
That seems rather more concerning than simply being corporate owned, and PGP signatures would not solve it.
The only real solution I see is to stop using PyPI entirely. Anything else would be the easy bullshit response. Are you willing to go that far? Probably not, since we're both on a YCombinator-hosted discussion forum.
122938677 7 hours ago [-]
For a long time, Google did not interfere or make demands of python core developers.
It started when Microsoft bought its way in, the Windows installer was taken away from Martin von Löwis, who soon left afterwards and the Microsoft people started to push for a GitHub move.
Then other companies followed to control developers more. There are also those who make large contributions and then disappear. They aren't independently wealthy either. It is the power grabbers and the mediocre who profit from it. The corporate takeover could not have succeeded if there had been a unified front.
> Are you willing to go that far?
Yes, I grab the original sources, sometimes from Debian .orig packages. I could also use Anaconda or something, which is still corporate but less egregious.
eesmith 5 hours ago [-]
Van Rossum worked at Google from 2005 to 2012. You think he completely avoided the structural bias to have Python solve problems Google had? Including collections.abc?
Very strange then that PEP 3119, with van Rossum as co-author, says "During the development of this PEP and of its companion, PEP 3141, we ..." and PEP 3141 is by a Google employee.
Why, it's almost as if my point was not at all about an employer interfering or making demand, but about aligned interests. As a rule, tobacco companies do not fund anti-smoker research, but also do not interfere or make demands, so those researcher can say they were not responding to pressure from their funders.
I remember the long discussion of the pros and cons of switching from Bitbucket to GitHub. The plan started in 2014, says PEP 512.
It's very clever of these Microsoft people to push for a move to GitHub some 4 years before Microsoft acquired GitHub in 2018. And to come up with a pretty good set of reasons to migrate, all documented, including a counter-proposal to use GitLab.
Almost as if it were a consensus-based decision to move, rather than a clever black-ops by Microsoft.
> I grab the original sources
Okay, then why do you care about this change or the use of PGP? Are you acting as an advocate for others, so that PyPI does not get worse for them?
In which case, why do you want them to continue to use a service where the only official way to get support is to have a Microsoft account? If you don't like PyPI, get more and more people to host their packages off PyPI.
The big gaping problem is Dependency Confusion Attacks, as in the rejected PEP 708.
Distribute your packages on a new server, using the Simple Repository API. Tell people to "pip install -i" to install your package. Take a look at the access log to see pip try to install other packages, because pip was designed for mirrors, not multiple hosting sites, each with their own potential namespace. There's no namespacing either.
From what I've read, that's pretty baked into pip, and will be hard to fix. It won't change until there are dozens of popular packages which have decided to avoid PyPI. In the meanwhile it will encourage even more people to use uv, which at least has a mechanism people can opt into.
woodruffw 1 days ago [-]
There's a discussion thread for email attestations here[1]. Those would support local builds.
But to take a step back: if you want attested builds because you think attestation is itself a gold star, then you're working from the wrong priors. Attestations do not imply that a package is secure any more than a hash does; what they are is a cryptographic signature over a set of claims, which allows for authenticated auditing of a project's state over time.
I think PyPI's documentation does a good job of conveying this nuance[2], and that much of the opprobrium of attestations comes from people either not understanding that nuance or imputing conspiracies into it.
> what they are is a cryptographic signature over a set of claims, which allows for authenticated auditing of a project's state over time.
Which is great--but does not in any way require that the attestation can only be produced by working through one of a small set of recognized "forges". You could do this just as easily by, for example, letting me upload to my PyPI account a public key (the same as forges like Github do to allow SSH access), and then letting me sign builds that I upload using the corresponding private key. But that kind of workflow is not being supported. Why not?
woodruffw 1 days ago [-]
> Which is great--but does not in any way require that the attestation can only be produced by working through one of a small set of recognized "forges"
That's true, there's some context elsewhere in these threads about how PyPI can enable email attestations. The short version is that PyPI wanted to make this work by default, which was trivial with Trusted Publishing (since it already provides a strong source of verifiable machine identity).
"Why not a public key?" is a good question, and the very short answer is that greenfield signing designs should not assume that all users are able to manage the secrecy of long-lived key materials, much less force them to make parameter decisions when (re)generating materials. This is the same mistake that the Web PKI made for decades, and has spent the last 15 years slowly untangling. The ideal setup is one where users demonstrate control over an identity, and then the signing scheme binds that identity to a short-lived signing key. That's how email attestations work.
pdonis 1 days ago [-]
> email attestations
From what I can gather from the github issue that was linked to, those would still require the email identity to be vouched for by a provider. (Though if whoever already is hosting my email could be a "provider", that would at least greatly expand the list of providers from the current small number of forges.) There doesn't seem to be any way for me to, for example, link a public key I generate myself to an email address that PyPI already has on file for me.
zahlman 20 hours ago [-]
> "Why not a public key?" is a good question, and the very short answer is that greenfield signing designs should not assume that all users are able to manage the secrecy of long-lived key materials, much less force them to make parameter decisions when (re)generating materials.
How is offering the public-key option in any way making such an assumption or applying any such force?
woodruffw 13 hours ago [-]
As can be seen in every sub-thread of this HN post, users are constantly buffeted by bad advice around key management and generation. It’s the reason applied cryptographers still have to even say “don’t use PGP,” despite it not being a sensible thing to even seek out in 2026 (much less 2016).
(Other large-scale PKI initiatives, like the Web PKI, learned this a long time ago. It’s why we have the CABF BRs instead of asking end users to pinky promise that they’re going to securely generate -‘d store a key and keep it around for 10+ years.)
pdonis 7 hours ago [-]
So basically, you are viewing an attestation as making some kind of promise about how carefully the attester handles their private keys. And you're willing to trust a small number of large tech companies to do that, but you're not willing to trust individuals like me to do it. Am I reading that right?
woodruffw 6 hours ago [-]
No, I wouldn't say that's right. Attestations are signatures from user-held keys under the hood; they're just short-lived keys bound to an identity rather than long-lived keys.
The nuance is that the party that does that binding is typically someone who is doing more than keeping their private keys on disk, which is why they're trusted to handle more than short-lived key materials. As mentioned elsewhere that can include "big tech companies" (is ActiveState big? I have no idea) but PyPI also supports self-hosted GitLab instances for organizations. Key management is what matters to us, and knowing that we can trust the parties that we still have to trust.
pdonis 4 hours ago [-]
> the party that does that binding
In other words, you only trust parties to do the binding whose key management you find to be acceptable, and "I keep my private key on my local machine that nobody else ever has access to" is not acceptable. Correct?
> PyPI also supports self-hosted GitLab instances for organizations
How does PyPI verify that those organizations are doing key management in a way you find acceptable?
pdonis 7 hours ago [-]
> the Web PKI
The Web allows self-signed certificates. I can put one on my website and people can use HTTPS to connect to it that way. And then it's up to each person who goes to my website to decide how secure my self-signed certificate is, as opposed to one that comes from a CA.
But of course the browser vendors don't look at it that way, they'll put serious warnings up to the user if they see a self-signed certificate, which they won't for a CA-signed certificate. That's basically your thought process as well, yes? Except that since you control the equivalent of the "website" and its certificate here (namely PyPI and its upload process), you won't even allow the equivalent of a self-signed certificate at all.
woodruffw 6 hours ago [-]
The Web PKI does not allow self-signed certificates. You can self-sign a certificate and badger your users into trusting it, but as you note the browsers won't help you do that because it's a security pit of despair.
> you won't even allow the equivalent of a self-signed certificate at all.
You can embed a signature into a Python package if you want. There's even a spec for it[1], although nobody has ever really adopted it. And, of course, you can always host a signature somewhere else (and even link to it via standard packaging metadata). But PyPI will no longer allow you to upload a PGP sidecar (which were never standardized, either) because we have strong evidence that (1) nobody ever really bothered to verify them, and (2) even if they had, the quality of the key material was poor.
> The Web PKI does not allow self-signed certificates.
That's not what the rest of your post says. The rest of your post says that you can use one for your website--which of course is true since no one can stop you if you insist on doing it--but that the rest of the Web infrastructure will treat your certificate poorly.
I am simply pointing out that it looks to me like PyPI's attestation mechanism is taking a similar viewpoint to the Web infrastructure in this respect.
> You can embed a signature into a Python package if you want.
I'm talking about the attestation mechanism. That mechanism, unless I'm missing something, does not support self-signed certificates.
eesmith 1 days ago [-]
Have you considered that perhaps you are working from the wrong priors?
I believe your prior is that you expect people who read the PyPI page to understand what attestation means. I believe your prior is they will see a notice about "not from a trusted publisher" and not treat it like demerit or warning. I believe your prior is that seeing this message will get people to do more due diligence than they currently do, and that the due diligence will be relevant and appropriate.
Based on my experience with my users, I don't have the same confidence.
I don't understand what "authenticated auditing" means, so why should they?
What project state do people think is important to audit? My code repo is private, hosted on Sourcehut. My build system is private, done locally. The only thing I release is the pre-built binary package. There's very little state to examine, and I can't figure out why anyone cares.
I also have customers who buy a source code license. They have never asked for an audit of the project's state over time.
I read the "Security model and considerations" document. I found it opaque, no doubt because the entire topic is foreign to me, but it doesn't do a good job of explaining. Take the second paragraph:
"The most basic type of attestation (the type enabled by default in pypa/gh-action-pypi-publish) asserts that the project was published by an authorized publisher such as a particular CI provider in a particular code repository.".
I know pypa/gh-action-pypi-publish is some sort of GitHub thing. I don't use GH or third-party CI so that clarification does not help.
It doesn't describe what "authorized publisher" means, other than a generic example of "particular CI provider in a particular code repository". That phrase exists only once in the document. Is it different from a "Trusted Publisher"?
Why are the terms "authorized" and "Trusted" used, instead of something like "affiliated"? If someone is not a trusted publisher or an authorized publisher it's is too easily misinterpreted as "untrusted publisher" or "unauthorized publisher", which seems scary. If someone is not an affiliated publisher, "unaffiliated publisher" sounds less scary. (I'm not saying "affiliated" is the right word, only an example of a roughly appropriate word where the negation of it is not so scary.)
Then later, "It protects against modification of a project after it was built".
I think that's supposed to be "modification of a release after it was built"?
My use of "project" matches the one at https://docs.pypi.org/using-pypi/pypi-basics/ "A project is a collection of releases and files, and information about them. ... A "release" on PyPI is a specific version of a project." I want to modify the project with updated metadata as things change, and to make new releases.
Oh, and I can't find a description of what "build" means in that document. It's used as if everyone expects to know what it means, but a few years ago I learned that modern Python packaging uses "build" different than I do.
When I compile my Python/C extension I "build" it. When I make a wheel I "package" it.
I believe that modern packaging calls the whole thing "build"? Or just the latter step?
Incidentally, "python -m build --sdist" on my project does not work as a replacement for "python setup.py sdist" because my setup.py includes a code generation step to get from a clean repo to the Python/C extension files included in the distribution. I haven't figured out the modern way to handle that. I think it's because of the differences in philosophy of what "build" means, and what actions are allowed during different stages of the distribution process, which is why I think "build" needs a specific definition, just like "project" and "release" have specific definitions.
woodruffw 1 days ago [-]
I mean, my actual prior here is that nobody should care or even think about this stuff. Authenticity and minimally-scoped credentials should be ambient properties of a system, not something that users even need to read docs for, much less enable. But if we're engaging with the premise that users are looking at this stuff at all, then I think it's facially reasonable to expect them to read the docs.
For context: attestations aren't something you even really enable. They just happen, by default, if you use Trusted Publishing from a forge like GitHub or GitLab (and hopefully, more in the future).
FWIW, your feedback on the docs is very helpful. It's genuinely useful to hear that the phrasing in it is imprecise, and I'll try and improve some of it this week. The main confusion that I think you're having (which is IMO reasonable) is that the doc was written to be theoretically splittable from the "Trusted Publishing" concept, i.e. so that "publishers" (principals who can publish) who aren't tied to a Trusted Publisher can attest to their published files.
pdonis 1 days ago [-]
> nobody should care or even think about this stuff.
Does that mean I, as someone who has projects on PyPI, shouldn't care if I am unable to provide any attestations because I don't want to have to go through one of the recognized forges?
> Authenticity and minimally-scoped credentials should be ambient properties of a system
But if the only way to get them is to go through one of the recognized forges, then they're not "ambient properties" of just PyPI alone, because there's no way for me to use them just with PyPI alone.
eesmith 1 days ago [-]
Okay, but if the information is on the project page then it's pretty likely that users will look at this stuff without reading the docs. My feeling was the doc was written for people "Publishing Projects to PyPI" and not for people "Using PyPI", quoting the relevant sections of the documentation.
Of course, in real life 99.9% of people aren't looking at the page at all, and either following a documentation step which says "install by doing XYZ" or transitively from some other project they install.
I'll compare "attestations aren't something you even really enable" with this from the email attestation proposal: "Another possibility here would be to have it be opt-in: maintainers can mark their already-verified emails as being eligible for attestations on PyPI, e.g. via a checkbox in the user settings." ;)
I also can't help but wonder who reads https://docs.pypi.org/publishing/publishing-to-pypi/ where it says "For instructions on how to package your Python project, please refer to the Python Packaging User Guide.", follows those steps including the one about using twine and TestPyPI, does all that, and then gets to "Publishing package distribution releases using GitHub Actions CI/CD workflows" and gives up, since twine works.
Or goes back to the pypi documentation and sees "To use an API token: Set your username to __token__" when I can't find anything like that at https://packaging.python.org/en/latest/guides/publishing-pac... , Those seem like describing two very different systems, speaking as someone with no experience with them at all, who learned to use twine because of the Python Packaging User Guide.
1299343 1 days ago [-]
[flagged]
tzlahp 1 days ago [-]
Of course: In principle we'd welcome a change, but there are technical difficulties in perpetuity.
woodruffw 1 days ago [-]
Storing your private keys somewhere other than a regular file on disk is not a perpetual technical difficulty. Please assume good faith.
Kwpolska 1 days ago [-]
Where's the good faith in demanding a non-profit buys a $10k+ HSM just to get a gold star from an OpenAI employee?
woodruffw 1 days ago [-]
I don't think we require a Trusted Publishing provider to use a physical HSM. As I've said in other venues, even a self-hosted Vault instance would almost certainly be fine. But storing private keys on disk is not, and we owe PyPI's users a higher standard of care than saying otherwise.
(I also don't hand out gold stars, and my employer does not direct my work on PyPI.)
zmgsabst 9 hours ago [-]
Your posts about how you’re restricting people for their own good make a lot more sense knowing you’re funded by OpenAI.
I think you should be more upfront about that funding — because while you may not perceive yourself as biased by that, others do.
woodruffw 9 hours ago [-]
My employer is a matter of public record, as is the fact that I’ve worked in PyPI for years on both my own time and at other jobs. You’re letting the tail wag the dog.
(Note also that none of the work being talked about right now was paid for by my current employer. I wasn’t even involved in it. I’m only familiar with it circa 2022-24, when I had a different employer.)
ghosty141 24 hours ago [-]
A yubikey is cheap and is good enough for that... No 10k jam necessary
eddythompson80 1 days ago [-]
Sounds like it. Currently it seems you'll need to setup your own OIDC provider[0], then work with PyPI to get them to trust your instance as a "Trusted Publisher". It seems like a system to scale for multi-tenant publishers (like millions of GitHub or GitLab or Google build or ActiveState users). The design itself should be able to scale for individual OIDC providers through webfinger and OIDC discovery like Tailscale does it[1], but it doesn't look like it's there yet. Not sure if it's planned either the github issue mentiones future PyPI verified email attestation, though I'm not sure if that implies webfinger/OIDC like ts does it.
> The design itself should be able to scale for individual OIDC providers through webfinger and OIDC discovery like Tailscale does it[1], but it doesn't look like it's there yet.
This is the right idea, but it's actually not that simple unfortunately: Trusted Publishing relies on per-IdP claims that are more precise than the `sub` claim (because doing pattern matching on the `sub` claim is brittle and prone to security problems). So we require publisher providers (OIDC IdPs) to provide distinct claims that we can match on. For GitHub and GitLab that's things like the repository name, workflow filename, etc., while for others it might be distinct username and immutable ID claims, etc.
This is all downstream of OIDC-driven federation being a much weaker and more bespoke process than all parties would like it to be. I think the discourse around this could be much improved by people taking the time to understand these complexities and why they pose a challenge to groups with limited engineering resources (like PyPI).
WhyNotHugo 1 days ago [-]
Provenance attestation is about security compliance, not real security. At a big organisation, you can check off compliance check-boxes because you're using Microsoft's big hosted service.
For regular developers, provenance makes no difference; whether the package was made in a box controlled by Microsoft with a proprietary stack or on a box controlled by the developer make no difference.
pasteleft 18 hours ago [-]
Are these security signals also available in pip??
I don't really look into PyPI package page before typing pip install numpy.
cowlevel 13 hours ago [-]
I only ever run "pip install" after getting the package name confirmed from some other source like the project's website. Historically it's been too easy for someone to hijack a similar package name with malicious code. For example is it "pip install requests" or "pip install python-requests"? Is it "pip install cryptography" or "pip install pycrypto"? At least historically, one of those was correct and the other one pwned your computer.
seekstars306 14 hours ago [-]
[flagged]
darkamaul 1 days ago [-]
Not directly the subject but they switched from Material for MkDocs to Zensical.
It’s quite sad to see that one of the main (or the biggest?) documentation system for the Python completely alienated its user base and led to this fork.
duskdozer 1 days ago [-]
Judging by the readme and homepage, there were probably some large differences brewing. Can't say I'm a fan of how it looks like they'll take the UI
zoobab 11 hours ago [-]
Is this security scanning actually open source?
s_u_d_o 1 days ago [-]
Great news! Would love to see provenance on Homebrew (brew) as well.
woodruffw 1 days ago [-]
Homebrew actually has full provenance. We added it in 2024 (I think, that’s from memory). We never flipped the default verification policy because of some integration concerns, but my understanding is that various downstreams and MDMs do consume and verify Homebrew’s attestations.
Edit: I should qualify: Homebrew’s own processes verify its attestations. But that’s not enabled by default on end user machines because it slows installations down and currently requires the `gh` CLI. Neither of those is a tenable long term constraint for a default feature.
let me tell you something incredibly important about the attestations / trusted publisher thing:
if you have any of your files published by a trusted publisher with attestations, then you must have all of your files published that way (in that package).
otherwise pypi determines your project may have been compromised and puts some big red warnings up like [1]. it does not try to look at "this particular source file was never part of trusted publishing" or anything like that. and, CRITICALLY, it does not send you an email or anything like that about the non-acceptable file that you just uploaded, which it REALLY should. I mean the model is, they think these uploads are malicious by an attacker. Emailing the project maintainer(s) would be pretty important! (When pypi wanted to get rid of GPG signatures, they started sending me an email for everysinglesignature that got uploaded. so i know you guys know how to send emails).
it's also AFAICT a permanent switch. once you put a single trusted/attested file up there, your account must now do all files in this way forever (or risk big red warnings. which maybe you're OK with that).
It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
> it's also AFAICT a permanent switch. once you put a single trusted/attested file up there, your account must now do all files in this way forever (or risk big red warnings. which maybe you're OK with that).
It's not[1].
> It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
As mentioned in several other places, we're always interested in adding more publisher providers, provided they meet our key management and notability requirements.
(This is also the easiest way to reason abductively about PyPI's goals. If PyPI's goal was to support "large for-profit corporations," we'd have dozens of publisher providers of dubious quality and community value, rather than the small subset we currently have.)
> This is also the easiest way to reason abductively about PyPI's goals. If PyPI's goal was to support "large for-profit corporations," we'd have dozens of publisher providers of dubious quality and community value, rather than the small subset we currently have.)
I never claimed pypi is trying to "support large for-profit corporations". I fully understand the "large corporations as trusted publisher" thing is because this particular subset, e.g. github, gitlab, google, are widely known institutions with "good" reputations (in this area). The issue is that pypi is not, as of yet, supporting individual projects that want trusted publishing without relying on a large for-profit corporation. I'm not too put off by it but it is definitely a thing that a private for-profit host overnight becomes horrible (see e.g. bitbucket) and I have to get off that company ASAP. Having some non-profit entities here would be extremely helpful.
woodruffw 1 days ago [-]
> Having some non-profit entities here would be extremely helpful.
I fully agree. I am hopeful that Forgejo or whatever other underlying component of Codeberg can improve its key management primitives here, which would make it very easy to argue for their inclusion as a federated party.
tzlahp 1 days ago [-]
You want the ones that employ core devs through Microsoft or are approved by OpenAI, which codeberg.org is not.
tzlahp 1 days ago [-]
This should be the top comment. PyPI as usual screws up and spreads misinformation about projects.
Thought we already learned years ago an IP address is not a person?
That is to say; what the screen on my end communicates is not guaranteed to be representative of ground truth.
The IT world was recently making fun of Sam Altman's eye scanner because remote attestation can be faked.
All the layers of abstraction wrapped around the code is security through obscurity.
The code itself must still be verified as safe.
The cloud era has resulted in a lot of people operating at layers of abstraction so far removed from physics this shit has more in common with baseless dogmas, gospels, and spiritualities at this point.
We anthropomorphized the cloud inserting ourselves into it. It's just a bunch of networked computers.
All this stuff is a spoon that doesn't exist.
bonoboTP 1 days ago [-]
You're speaking in such shorthand that I have no idea how to untangle what you're actually trying to say/propose. I suppose you're saying that attested code can still be malware because malicious actors can also go through the attestation process?
rcxdude 1 days ago [-]
Yeah, this check basically just proves that you hosted your source code on a big platform and then built the pypi package with their CI. This is not particularly strong as a trust signal. Especially since it doesn't particularly protect against compromise of access to the code repository. I worry that this is going to distract from better approaches (like reproducable builds and/or distributed review).
abcmall 1 days ago [-]
[flagged]
tdeck 1 days ago [-]
Two days seems like a decent response time to make an improvement, no?
jlak-aski 1 days ago [-]
[flagged]
abbadadda 1 days ago [-]
What’s the story? I seem to have missed this.
neuronexmachina 1 days ago [-]
Yeah, I have no idea what they're talking about either.
The drama here is that 2 years ago, a Python core dev was suspended for 3 months as part of a mailing list fight?
aftbit 1 days ago [-]
I'm also out of the loop, but from my reading, I guess some people didn't like Tim Peters and his positions on some culture war stuff, and didn't like that people were generally focusing on his positive contributions to the project right after his death, rather than his views on race or whatever.
akerl_ 1 days ago [-]
Oh, that's even weirder. So the drama was that people were saying nice things about somebody who had just passed?
Then I'm more confused than before about why the original commenter on this branch would suggest this security feature release is a "diversion". A diversion from what? The fact that people weren't talking about something?
1 days ago [-]
Rendered at 00:18:49 GMT+0000 (Coordinated Universal Time) with Vercel.
Did I get that right? Seems a bit anti-pythonic.
That way projects that want to be extra safe can set up multiple providers to check the code independently, and third parties can upload their own attestation that PyPI doesn't trust by default but you can add them to your private whitelist. And they wouldn't have to care about where the release is uploaded from.
Crypto projects know that 1 of 1 is the worst possible trust assumption. If you want real confidence you need more than one party.
(Python packaging isn’t anywhere close to reproducible, and it isn’t clear how PyPI would enforce that without pissing off >90% of the ecosystem.)
The growth of centralized authorities arises from the reality that the vast majority of devs do not and will not independently vet or attest to the quality of libraries they use. That’s not great, but it is what we have and doesn’t look likely to change.
Given that, and given that people still want as much security as they can conveniently get, centralized control (from “we run a security scanner and trust its findings” to “packages can only be downloaded from pypi/debian stable/whatever”) is a natural result.
I'd be wary of trying to estimate the split of opinions based on who comments.
(I've had a quick read through the various links but they mostly seem to emphasize what it's not, which is why I say that it doesn't seem to improve security, and especially I don't really see a motivation for why these other parties need to be involved in the way they are: the only benefit I can imagine is a third-party attestation that some artifact corresponds to some source code, which seems to be partly besides the point and partly solvable through better means)
https://news.ycombinator.com/item?id=50499144
https://news.ycombinator.com/item?id=50500622
https://news.ycombinator.com/item?id=50499218
If you're curious to dig into what the checkmark means, why you should care, and why they implemented it, I'm not going to be able to do a better job than https://docs.pypi.org/attestations/security-model/
Here's some extra context [0]: https://github.com/pypi/warehouse/issues/20927. Loooots of links to click in.
the point here being small service providers are _also_ able to get into this system, but I think there's an expectation of success.
The bar for trusted publishers is very high though. I don't think it's high "on purpose" to leave out certain platforms for being too small, but in effect if (for example) your provider doesn't have a security contact point then that's going to rule them out here.
sans client verification of signatures by default, install is still roulette, and I’m glad package managers have introduced package publication date based cool off options now.
(Personally I think that we just shouldn't have automated releases via CI, full stop. There should "always" be a person pressing a button to release things. So I'm in a bit of a big tent with manual upload support. At least some second factor for releases!)
I think generally though the idea is that trusted publisher signing + upload is about simplifying .... something. But given how attacks have been happening in the past 24 months I fear that this might not help as much as we want.
Quite the opposite. "There should be one-- and preferably only one --obvious way to do it".
As a passionate anarchosyndicalist, I, for one, am thrilled about it. Decentralization is a tool in our toolbox for identifying and dismantling unjust hierarchies, but it is not a panacea, and can be quite damaging when taken to the extreme. See: cryptocurrency speedrunning 200y of bank regulations in 20y.
To put my trust for Python in more concrete terms: they support GitLab. As a passionate GitLab shill, I really couldn't be more impressed! It continues to shock me how infrequently the second most popular forge is acknowledged, much less supported. This shows they're not just, IDK, bowing to Microsoft's demands.
Not only don't we, but I must emphasize that Microsoft did not pay for anything involving the implementation of either Trusted Publishing or attestations on PyPI. I wrote a personal blog post explaining some of my own rationale for why we prioritized GitHub and GitLab for PyPI's initial support targets, and it shouldn't be surprising to anybody who knows me that the reasons are pragmatic, not dogmatic[1].
[1]: https://blog.yossarian.net/2024/11/18/Security-means-securin...
If not--i.e., if only recognized "forges" are ever going to be supported--why not?
Absolutely not claiming its not a pragmatic step, but equal-footing alternatives would be useful.
I'm no tin-foil-hat - but when a website account requires government id (like in the uk), and creating software requires signing by the website, you link all the things - is that a good outcome?
AnnasArchive in particular is a decent example to keep in mind, since she seems to host her own GitLab instance for opensource development. Obviously that doesn't solve this exact problem b/c self-hosted instances don't work as trusted publishers, but it speaks to what I've clumisly been trying to, here: when it comes to protecting software & speech that the government is trying to ban, making sure those people can host their own stuff and still access the minimum shared infrastructure to allow people to find and use/read it is a lot more practical than trying to build the 99%'s platforms around criminality.
Cause that's what we're talking about, here, to be clear. Criminality. Again I love some criminals and even some crimes, but the word's typical connotations seem like a pretty good reminder of what prioritizing such usecases in the mainstream can cost us.
https://openid.net/foundation/sponsoring-members/
We know why PyPI ripped out PGP support and has refused any verification mechanisms that other distributions use for decades.
Seems more straightforward than custom stuff built on OIDC, but I'm not a security guy, so maybe I'm missing something?
[0] - https://developers.yubico.com/PGP/Attestation.html [1] - https://docs.yubico.com/software/yubikey/tools/ykman/OpenPGP...
You can also look to FIDO2 / passkeys to see that a lot of the folks who have concerns about PyPI trusted publishing also view device attestation as a scourge.
The fact that a large company can mess things up doesn't change the fact that it's way more viable to review how Github or Gitlab or Codeberg handles key material and security than it is to make a comparable local developer flow that avoids common mistakes like just dropping the signing key on disk on whatever servers you want to run your builds from.
Only if you already trust them.
> dropping the signing key on disk on whatever servers you want to run your builds from
I run my builds on my local machine. The private key I would use to sign them if PyPI supported that is not anywhere in the repo I'm using.
Of course you would have to trust that I'm telling the truth in what I just said. But, as above, you also need to trust what the large tech forges are telling you in order to review how they handle their keys.
The repos workflow source, yes.
But the workflow engine implementation? I don't know how much of it is open source, but I am sure nobody external can audit what Microsoft really runs there. Another company starting with M has just been fined for accessing users' photos. So I would not be 100% surprised if Microsoft changed/amended some user code before executing it or interpreted it in an unpublished way e. g. "for security reasons"
2. I'm not entirely sure what the link is supposed to prove -- would appreciate a word of clarification if you find the time. It almost feels like a dogwhistle of some kind, but A) I doubt anyone on HN is anywhere near that problematic, and B) there's no clear pattern in those companies other than "interested in ID tech", and none of the usual dogwhistle targets made the list. Also, y'know, OpenAI isn't on it...?
I guess my fundamental confusion isn't about the list and more about what "we know". Are you implying that they are against pure, unmanaged PGP for this system because... I mean, why? Because they're playing the long game to promote a verification/attestation/identification (?) technology that you think offers less stringent privacy guarantees, and thus to sell more ads in 2036?
If so, doesn't that seem like a kinda roundabout way to raise Display Ads revenue, otherwise known as "that one dying industry"?
Anyway, sorry to confuse/frustrate, I was being honest and tried to explicitly not accuse you of dogwhistling. I was making a best-effort attempt to explain my confusion, rather than just replying with "that doesn't make any sense" or something.
I guess I'll briefly elaborate just in case: "dogwhistle" above refers to a specific kind of antisemetic practice where people point at a list of companies or institutions or even just people and say something like "what do you notice?", whith the message ofc being that they're all related-to/run-by Jewish people. Again, not accusing you of that -- those people don't generally come here. But absent any other logic, I am left with confusion.
I will briefly add that this very response also doesn't help, when you very clearly imply that I am "shooting down a comment" in bad faith for some sinister purpose. It, again, comes off as conspiratorial. Why would I --a sad 30 y/o child with my real name and phone number in my bio-- be shooting this down?
I guess my overall response is that this feels like a conspiracy theory without identifying the conspiracy.
dlt (no provenance): https://pypi.org/project/dlt/#files
datasette (yes provenance): https://pypi.org/project/datasette/#files
[1]: https://github.com/pypi/warehouse/issues/new?template=bug-re...
I'm bought in on keyless signing, and I want there to be many more providers, so I like to see more adoption which builds more momentum.
How is a third party service, especially a self-hosted one, expected to meet those requirements? They seem to be lengthy, onerous, and also require a detailed vetting by the pypi team that is unlikely to scale beyond a few large hosting providers.
(What's more, this does relatively little to actually signal trust to someone downloading them. All it does is provide a signature that one of the big providers has built the package from a given source (but nothing about the trustworthiness of that source). This is a problem much better solved by reproducable builds and distributed verification of those builds)
Codeberg: https://codeberg.org/mfenniak/oidc-jwt Tangled: https://tangled.org/docs/spindles/workflows/#audience
I think OIDC tokens from workflows is basically table stakes at this point, since the alternative is a static secret, which is basically guaranteed to leak eventually.
All that's left is for PyPI to accept these tokens in their set of trusted publishers.
(I haven't looked at Tangled at all. If Tangled has a secret management story for its PKI that's stronger than a private key on disk and can demonstrate that a non-trivial number of PyPI users would benefit from federation, then it's a strong candidate.)
PKI definitely is still hard, but if it can be done well by a smallish number of trusted parties to the benefit of their users, this can have huge ecosystem benefits. Let's hope we live in that timeline.
Inherently, if someone steals your key, or gains access to a box that can use that key for signing, they are indistinguishable from you. Hand PGP signing also lacks time-bound signing, any ratchet mechanism, CT log, consistent key hierarchy, or standard to bind signer name and project identity.
In the sense that I'm not a large corporation and nobody else is auditing my personal computer, yes, that's true. Note, though, that large corporations are also subject to nefarious actors misusing their infrastructure. What if a rogue employee hacks their internal system? Or Russia or China manages to get a backdoor into it?
In any case, all this doesn't make the method I described impossible. It just means that attestations produced by an individual workflow need to be clearly labeled to distinguish them from attestations produced using a tech forge's workflow, so people will understand what, specifically, is being attested to. Then people could weigh each trust signal as they choose.
> Hand PGP signing
To be clear, I wasn't talking specifically about this method of signing with a personal key. I agree there are lots of issues with PGP as an overall system. I was only talking about the general idea of having some way for individuals to independently generate an attestation without having to go through a tech forge's workflow.
We definitely should have a system that has multiple mechanisms for authorship attestation, of ongoing identity control, code control and binary generation sanctity. The last step can be made repeatable if we can manage reproducible builds.
Like, if I want to host on Codeberg, or Sourcehut, what is the process needed for those platforms to become a trusted publisher?
As it happens, I build my releases on my own hardware using a pypa/manylinux release. Can I provide attestation, or am I required to use a third-party service to get that "x Without provenance" removed from my entry?
(If you pan through that thread, you'll note that one of open issues with Codeberg is that their underlying server wants to store OIDC private keys on disk at the moment. This is okay for small deployments of personal services, but not for something that PyPI needs to federate with and trust.)
[1]: https://github.com/pypi/warehouse/issues/19786
Will PyPI be presenting warnings or statements of caution which might cause users to be concerned about using my distributions? Or imply that my build practices are somehow inferior to depending on a third-party build system?
Do you see any other alternative?
Both of these are latent problems when trying to use PGP in packaging ecosystems; the subset of ecosystems that successfully adopt PGP tend to have very different topologies (namely, small rings of trusted...publishers) rather than an "anyone can publish" architecture like PyPI has.
But that is not done, because python-dev is essentially corporate owned and has always done the bidding of corporations for at least 10 years because their "leaders" want to keep their easy bullshit jobs.
Those tend to be companies with a lot of money. Those companies tend to have different needs and preferences than a sole proprietor ISV like my company. They tend to hire people who are aligned with their goals. They tend to fund specific PSF projects which are aligned with those goals. Those with strong objections to those goals, or the source of the funding, don't work for those companies, quit, or get fired.
As it is, the entire PyPI infrastructure depends on the corporate largess from the likes of Fastly, Amazon, Microsoft and Google - the latter three of which are described in "From economy of occupation to economy of genocide", Report of the Special Rapporteur on the situation of human rights in the Palestinian territories occupied since 1967, as corporations profiting from the displacement and genocide of Palestinians in Gaza, in breach of international law.
That seems rather more concerning than simply being corporate owned, and PGP signatures would not solve it.
The only real solution I see is to stop using PyPI entirely. Anything else would be the easy bullshit response. Are you willing to go that far? Probably not, since we're both on a YCombinator-hosted discussion forum.
It started when Microsoft bought its way in, the Windows installer was taken away from Martin von Löwis, who soon left afterwards and the Microsoft people started to push for a GitHub move.
Then other companies followed to control developers more. There are also those who make large contributions and then disappear. They aren't independently wealthy either. It is the power grabbers and the mediocre who profit from it. The corporate takeover could not have succeeded if there had been a unified front.
> Are you willing to go that far?
Yes, I grab the original sources, sometimes from Debian .orig packages. I could also use Anaconda or something, which is still corporate but less egregious.
Very strange then that PEP 3119, with van Rossum as co-author, says "During the development of this PEP and of its companion, PEP 3141, we ..." and PEP 3141 is by a Google employee.
Why, it's almost as if my point was not at all about an employer interfering or making demand, but about aligned interests. As a rule, tobacco companies do not fund anti-smoker research, but also do not interfere or make demands, so those researcher can say they were not responding to pressure from their funders.
I remember the long discussion of the pros and cons of switching from Bitbucket to GitHub. The plan started in 2014, says PEP 512.
It's very clever of these Microsoft people to push for a move to GitHub some 4 years before Microsoft acquired GitHub in 2018. And to come up with a pretty good set of reasons to migrate, all documented, including a counter-proposal to use GitLab.
Almost as if it were a consensus-based decision to move, rather than a clever black-ops by Microsoft.
> I grab the original sources
Okay, then why do you care about this change or the use of PGP? Are you acting as an advocate for others, so that PyPI does not get worse for them?
In which case, why do you want them to continue to use a service where the only official way to get support is to have a Microsoft account? If you don't like PyPI, get more and more people to host their packages off PyPI.
The big gaping problem is Dependency Confusion Attacks, as in the rejected PEP 708.
Distribute your packages on a new server, using the Simple Repository API. Tell people to "pip install -i" to install your package. Take a look at the access log to see pip try to install other packages, because pip was designed for mirrors, not multiple hosting sites, each with their own potential namespace. There's no namespacing either.
From what I've read, that's pretty baked into pip, and will be hard to fix. It won't change until there are dozens of popular packages which have decided to avoid PyPI. In the meanwhile it will encourage even more people to use uv, which at least has a mechanism people can opt into.
But to take a step back: if you want attested builds because you think attestation is itself a gold star, then you're working from the wrong priors. Attestations do not imply that a package is secure any more than a hash does; what they are is a cryptographic signature over a set of claims, which allows for authenticated auditing of a project's state over time.
I think PyPI's documentation does a good job of conveying this nuance[2], and that much of the opprobrium of attestations comes from people either not understanding that nuance or imputing conspiracies into it.
[1]: https://github.com/pypi/warehouse/issues/18007
[2]: https://docs.pypi.org/attestations/security-model/#trustwort...
Which is great--but does not in any way require that the attestation can only be produced by working through one of a small set of recognized "forges". You could do this just as easily by, for example, letting me upload to my PyPI account a public key (the same as forges like Github do to allow SSH access), and then letting me sign builds that I upload using the corresponding private key. But that kind of workflow is not being supported. Why not?
That's true, there's some context elsewhere in these threads about how PyPI can enable email attestations. The short version is that PyPI wanted to make this work by default, which was trivial with Trusted Publishing (since it already provides a strong source of verifiable machine identity).
"Why not a public key?" is a good question, and the very short answer is that greenfield signing designs should not assume that all users are able to manage the secrecy of long-lived key materials, much less force them to make parameter decisions when (re)generating materials. This is the same mistake that the Web PKI made for decades, and has spent the last 15 years slowly untangling. The ideal setup is one where users demonstrate control over an identity, and then the signing scheme binds that identity to a short-lived signing key. That's how email attestations work.
From what I can gather from the github issue that was linked to, those would still require the email identity to be vouched for by a provider. (Though if whoever already is hosting my email could be a "provider", that would at least greatly expand the list of providers from the current small number of forges.) There doesn't seem to be any way for me to, for example, link a public key I generate myself to an email address that PyPI already has on file for me.
How is offering the public-key option in any way making such an assumption or applying any such force?
(Other large-scale PKI initiatives, like the Web PKI, learned this a long time ago. It’s why we have the CABF BRs instead of asking end users to pinky promise that they’re going to securely generate -‘d store a key and keep it around for 10+ years.)
The nuance is that the party that does that binding is typically someone who is doing more than keeping their private keys on disk, which is why they're trusted to handle more than short-lived key materials. As mentioned elsewhere that can include "big tech companies" (is ActiveState big? I have no idea) but PyPI also supports self-hosted GitLab instances for organizations. Key management is what matters to us, and knowing that we can trust the parties that we still have to trust.
In other words, you only trust parties to do the binding whose key management you find to be acceptable, and "I keep my private key on my local machine that nobody else ever has access to" is not acceptable. Correct?
> PyPI also supports self-hosted GitLab instances for organizations
How does PyPI verify that those organizations are doing key management in a way you find acceptable?
The Web allows self-signed certificates. I can put one on my website and people can use HTTPS to connect to it that way. And then it's up to each person who goes to my website to decide how secure my self-signed certificate is, as opposed to one that comes from a CA.
But of course the browser vendors don't look at it that way, they'll put serious warnings up to the user if they see a self-signed certificate, which they won't for a CA-signed certificate. That's basically your thought process as well, yes? Except that since you control the equivalent of the "website" and its certificate here (namely PyPI and its upload process), you won't even allow the equivalent of a self-signed certificate at all.
> you won't even allow the equivalent of a self-signed certificate at all.
You can embed a signature into a Python package if you want. There's even a spec for it[1], although nobody has ever really adopted it. And, of course, you can always host a signature somewhere else (and even link to it via standard packaging metadata). But PyPI will no longer allow you to upload a PGP sidecar (which were never standardized, either) because we have strong evidence that (1) nobody ever really bothered to verify them, and (2) even if they had, the quality of the key material was poor.
[1]: https://peps.python.org/pep-0427/#signed-wheel-files
That's not what the rest of your post says. The rest of your post says that you can use one for your website--which of course is true since no one can stop you if you insist on doing it--but that the rest of the Web infrastructure will treat your certificate poorly.
I am simply pointing out that it looks to me like PyPI's attestation mechanism is taking a similar viewpoint to the Web infrastructure in this respect.
> You can embed a signature into a Python package if you want.
I'm talking about the attestation mechanism. That mechanism, unless I'm missing something, does not support self-signed certificates.
I believe your prior is that you expect people who read the PyPI page to understand what attestation means. I believe your prior is they will see a notice about "not from a trusted publisher" and not treat it like demerit or warning. I believe your prior is that seeing this message will get people to do more due diligence than they currently do, and that the due diligence will be relevant and appropriate.
Based on my experience with my users, I don't have the same confidence.
I don't understand what "authenticated auditing" means, so why should they?
What project state do people think is important to audit? My code repo is private, hosted on Sourcehut. My build system is private, done locally. The only thing I release is the pre-built binary package. There's very little state to examine, and I can't figure out why anyone cares.
I also have customers who buy a source code license. They have never asked for an audit of the project's state over time.
I read the "Security model and considerations" document. I found it opaque, no doubt because the entire topic is foreign to me, but it doesn't do a good job of explaining. Take the second paragraph:
"The most basic type of attestation (the type enabled by default in pypa/gh-action-pypi-publish) asserts that the project was published by an authorized publisher such as a particular CI provider in a particular code repository.".
I know pypa/gh-action-pypi-publish is some sort of GitHub thing. I don't use GH or third-party CI so that clarification does not help.
It doesn't describe what "authorized publisher" means, other than a generic example of "particular CI provider in a particular code repository". That phrase exists only once in the document. Is it different from a "Trusted Publisher"?
Why are the terms "authorized" and "Trusted" used, instead of something like "affiliated"? If someone is not a trusted publisher or an authorized publisher it's is too easily misinterpreted as "untrusted publisher" or "unauthorized publisher", which seems scary. If someone is not an affiliated publisher, "unaffiliated publisher" sounds less scary. (I'm not saying "affiliated" is the right word, only an example of a roughly appropriate word where the negation of it is not so scary.)
Then later, "It protects against modification of a project after it was built".
I think that's supposed to be "modification of a release after it was built"?
My use of "project" matches the one at https://docs.pypi.org/using-pypi/pypi-basics/ "A project is a collection of releases and files, and information about them. ... A "release" on PyPI is a specific version of a project." I want to modify the project with updated metadata as things change, and to make new releases.
Oh, and I can't find a description of what "build" means in that document. It's used as if everyone expects to know what it means, but a few years ago I learned that modern Python packaging uses "build" different than I do.
When I compile my Python/C extension I "build" it. When I make a wheel I "package" it.
I believe that modern packaging calls the whole thing "build"? Or just the latter step?
Incidentally, "python -m build --sdist" on my project does not work as a replacement for "python setup.py sdist" because my setup.py includes a code generation step to get from a clean repo to the Python/C extension files included in the distribution. I haven't figured out the modern way to handle that. I think it's because of the differences in philosophy of what "build" means, and what actions are allowed during different stages of the distribution process, which is why I think "build" needs a specific definition, just like "project" and "release" have specific definitions.
For context: attestations aren't something you even really enable. They just happen, by default, if you use Trusted Publishing from a forge like GitHub or GitLab (and hopefully, more in the future).
FWIW, your feedback on the docs is very helpful. It's genuinely useful to hear that the phrasing in it is imprecise, and I'll try and improve some of it this week. The main confusion that I think you're having (which is IMO reasonable) is that the doc was written to be theoretically splittable from the "Trusted Publishing" concept, i.e. so that "publishers" (principals who can publish) who aren't tied to a Trusted Publisher can attest to their published files.
Does that mean I, as someone who has projects on PyPI, shouldn't care if I am unable to provide any attestations because I don't want to have to go through one of the recognized forges?
> Authenticity and minimally-scoped credentials should be ambient properties of a system
But if the only way to get them is to go through one of the recognized forges, then they're not "ambient properties" of just PyPI alone, because there's no way for me to use them just with PyPI alone.
Of course, in real life 99.9% of people aren't looking at the page at all, and either following a documentation step which says "install by doing XYZ" or transitively from some other project they install.
I'll compare "attestations aren't something you even really enable" with this from the email attestation proposal: "Another possibility here would be to have it be opt-in: maintainers can mark their already-verified emails as being eligible for attestations on PyPI, e.g. via a checkbox in the user settings." ;)
I also can't help but wonder who reads https://docs.pypi.org/publishing/publishing-to-pypi/ where it says "For instructions on how to package your Python project, please refer to the Python Packaging User Guide.", follows those steps including the one about using twine and TestPyPI, does all that, and then gets to "Publishing package distribution releases using GitHub Actions CI/CD workflows" and gives up, since twine works.
Or goes back to the pypi documentation and sees "To use an API token: Set your username to __token__" when I can't find anything like that at https://packaging.python.org/en/latest/guides/publishing-pac... , Those seem like describing two very different systems, speaking as someone with no experience with them at all, who learned to use twine because of the Python Packaging User Guide.
(I also don't hand out gold stars, and my employer does not direct my work on PyPI.)
I think you should be more upfront about that funding — because while you may not perceive yourself as biased by that, others do.
(Note also that none of the work being talked about right now was paid for by my current employer. I wasn’t even involved in it. I’m only familiar with it circa 2022-24, when I had a different employer.)
[0] https://docs.pypi.org/trusted-publishers/
[1] https://tailscale.com/docs/integrations/identity/custom-oidc
This is the right idea, but it's actually not that simple unfortunately: Trusted Publishing relies on per-IdP claims that are more precise than the `sub` claim (because doing pattern matching on the `sub` claim is brittle and prone to security problems). So we require publisher providers (OIDC IdPs) to provide distinct claims that we can match on. For GitHub and GitLab that's things like the repository name, workflow filename, etc., while for others it might be distinct username and immutable ID claims, etc.
This is all downstream of OIDC-driven federation being a much weaker and more bespoke process than all parties would like it to be. I think the discourse around this could be much improved by people taking the time to understand these complexities and why they pose a challenge to groups with limited engineering resources (like PyPI).
For regular developers, provenance makes no difference; whether the package was made in a box controlled by Microsoft with a proprietary stack or on a box controlled by the developer make no difference.
I don't really look into PyPI package page before typing pip install numpy.
It’s quite sad to see that one of the main (or the biggest?) documentation system for the Python completely alienated its user base and led to this fork.
Edit: I should qualify: Homebrew’s own processes verify its attestations. But that’s not enabled by default on end user machines because it slows installations down and currently requires the `gh` CLI. Neither of those is a tenable long term constraint for a default feature.
brew install gh echo HOMEBREW_VERIFY_ATTESTATIONS=1 >> $HOME/.zshrc brew verify gh
Downloading attestations will be quicker if you are logged in to github: gh auth login gh auth status
For a box you don't want full login privs on, you can use a GH PAT token with minimal privs.
If too slow/GitHub down:
HOMEBREW_NO_VERIFY_ATTESTATIONS=1 brew upgrade
https://docs.brew.sh/Homebrew-Security-and-Supply-Chain
if you have any of your files published by a trusted publisher with attestations, then you must have all of your files published that way (in that package).
otherwise pypi determines your project may have been compromised and puts some big red warnings up like [1]. it does not try to look at "this particular source file was never part of trusted publishing" or anything like that. and, CRITICALLY, it does not send you an email or anything like that about the non-acceptable file that you just uploaded, which it REALLY should. I mean the model is, they think these uploads are malicious by an attacker. Emailing the project maintainer(s) would be pretty important! (When pypi wanted to get rid of GPG signatures, they started sending me an email for every single signature that got uploaded. so i know you guys know how to send emails).
it's also AFAICT a permanent switch. once you put a single trusted/attested file up there, your account must now do all files in this way forever (or risk big red warnings. which maybe you're OK with that).
It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
[1] https://fosstodon.org/@zzzeek/117554513935692558
It's not[1].
> It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
As mentioned in several other places, we're always interested in adding more publisher providers, provided they meet our key management and notability requirements.
(This is also the easiest way to reason abductively about PyPI's goals. If PyPI's goal was to support "large for-profit corporations," we'd have dozens of publisher providers of dubious quality and community value, rather than the small subset we currently have.)
[1]: https://discuss.python.org/t/pypis-new-security-indicators-a...
I never claimed pypi is trying to "support large for-profit corporations". I fully understand the "large corporations as trusted publisher" thing is because this particular subset, e.g. github, gitlab, google, are widely known institutions with "good" reputations (in this area). The issue is that pypi is not, as of yet, supporting individual projects that want trusted publishing without relying on a large for-profit corporation. I'm not too put off by it but it is definitely a thing that a private for-profit host overnight becomes horrible (see e.g. bitbucket) and I have to get off that company ASAP. Having some non-profit entities here would be extremely helpful.
I fully agree. I am hopeful that Forgejo or whatever other underlying component of Codeberg can improve its key management primitives here, which would make it very easy to argue for their inclusion as a federated party.
Thought we already learned years ago an IP address is not a person?
That is to say; what the screen on my end communicates is not guaranteed to be representative of ground truth.
The IT world was recently making fun of Sam Altman's eye scanner because remote attestation can be faked.
All the layers of abstraction wrapped around the code is security through obscurity.
The code itself must still be verified as safe.
The cloud era has resulted in a lot of people operating at layers of abstraction so far removed from physics this shit has more in common with baseless dogmas, gospels, and spiritualities at this point.
We anthropomorphized the cloud inserting ourselves into it. It's just a bunch of networked computers.
All this stuff is a spoon that doesn't exist.
Then I'm more confused than before about why the original commenter on this branch would suggest this security feature release is a "diversion". A diversion from what? The fact that people weren't talking about something?