Hypothesis: a big accelerant of these rapid repository compromise (from Red hat to GitHub to Amazon to small startups) might be GitHub+dependabot automatic dependency updates.
So, just like COVID-19 used air travel, modern malware attacks are relying on GitHub+dependabot to speed up the spread.
Even for single page website built using Vue, I would get about 5 updates a week.
eranation 17 hours ago [-]
So. Dependabot (and renovate) do have "cooldowns" supported, just need to set them up.
For dependabot it's as simple as
cooldown.default-days: 1
There are security researchers (that don't have cooldowns) that usually detect compromises within hours or less, and package managers almost always manage to remove the offending versions in less than 24 hours (usually much less).
So people will 24 hours cooldowns get protected.
Shameless plug: I maintain depsguard.com that tries to simplify cooldowns setup across anything that supports it, in one command (it scans from where you run it, e.g. if you run it from your user folder it will look for any local repos with dependabot / renovate and suggest a change.
ashishb 14 hours ago [-]
> For dependabot it's as simple as
> cooldown.default-days: 1
Most people stick to default of 0.
In fact, I am realizing over time that it is best to make it 7-14 days.
woodruffw 20 hours ago [-]
It’s plausible. It’s certainly the case that we (meaning security practitioners) spent years trying to move people onto faster and more automated update cycles, and these kinds of compromises have revealed a latent weakness that comes wit doing so.
swader999 1 days ago [-]
Aren't we back to the drawing board once everyone uses this?
grncdr 1 days ago [-]
I think the idea is that dedicated security firms and/or automated scanners will discover exploits in the cooldown period.
woodruffw 1 days ago [-]
Yep, this is the thesis behind them. I wish people engaged more fully with this argument: it’s possible to believe that security vendors won’t do a good job of upholding their side of the bargain, but I’ve yet to see anybody argue that rather than making a faulty universalization argument against cooldowns.
rendaw 16 hours ago [-]
How many people are paying security firms to test these things during the cooldown period? Which security firms are testing which packages?
woodruffw 16 hours ago [-]
> How many people are paying security firms to test these things during the cooldown period?
More than I think should be, frankly. More than enough for a sustainable industry.
As for “which firms”: if you Google any of the recent dependency compromises, you’ll see their names. My rough guess is that there are somewhere between 12 and 20 active players in the “supply chain security” space, and they generally compete for mindshare with blogspam. That’s not to say their scanning results aren’t good, though.
mike_hock 20 hours ago [-]
Anything that isn't blatantly obvious zero effort malware is gonna wait for the cooldown to expire and then gradually introduce backdoors or vulnerabilities in subsequent versions once the project has gained trust.
woodruffw 20 hours ago [-]
I think you’re talking about something different than the median case here: the kinds of malware that cooldowns are effective against are attacking already trusted projects, e.g. via repository compromise.
I think it remains to be seen whether the economics of this kind of “supply chain” compromise support a stealthier actor profile: the whole reason to compromise an OSS project rather than exploit a single victim’s browser or phone is to smash-and-grab en masse. That’s a fundamentally noisy and delay-sensitive attacker profile.
weinzierl 1 days ago [-]
If this is the idea, why don't we let the dedicated security firms and/or automated scanners find the vulnerabilities before the release?
You need an early release in the "given enough eyeballs all bugs are shallow" world because you need the eyeballs, but if you count on specialists and scanners no general availability release is necessary and hence no cool down.
nixpulvis 22 hours ago [-]
Because this isn't about guarding against general vulnerabilities. This is about guarding against a class of supply chain attack where an attacker compromises the packaging system in a way that they slip in a malicious release.
It is true if everyone waits a cooldown (I don't like this name, but that's another thing) period, then it doesn't solve anything, but as others have pointed out, just because you don't build off the cooldown doesn't mean you and others aren't watching the releases.
Closing the window makes the burden of hiding the exploit higher, which is I think objectively an improvement of security posture.
zem 8 hours ago [-]
the we/you are different in each case. say I maintain some package, foo, and I want to publish foo-3.5 which contains a security issue. in the cooldown scenario I push the new version to the package repository (rubygems/pip/npm/whatever), and the package manager when asked to `install foo` continues reporting 3.4 as the latest version until the cooldown period has passed. this happens automatically for all packages and all end users. simultaneously, a security firm could look at the log of new package submissions and take a closer look at anything that triggers their "this is an important/widely used package" filter, and if they find something during the cooldown period the new release could be removed from the repository without anyone having installed it.
in the "find the vulnerability before release" scenario, how do I push version foo-3.5? is it my responsibility as the package author to find a security firm and have them audit it before submitting to the repository? should the org owning the repository shunt my package update to a holding pen until it can be vetted, and if so do they have to pay a designated security firm and give them special access to the holding pen, or can anyone reads the incoming packages and help vet them? and if no one has looked at a package in a while do you delay it indefinitely or so you let it time out and be released? that's basically back to the cooldown idea, only implemented on the server rather than the client, and therefore not letting individual clients tune what they believe is a good cooldown time.
john_strinlai 24 hours ago [-]
i am not sure what the benefits of your proposal are compared to the "cooldown period" way.
the releases will be delayed for the same time period, but you increase the amount of coordination required significantly and reduce user agency.
weinzierl 22 hours ago [-]
Currently we release software to the public early and let people find vulnerabilities after release. A few early updaters get burned but the rest of us is saved. This cannot go on.
One idea to get out of this mess is to use cooldowns. But this idea only works if what GP said is true: Vulnerabilities are mostly found by (paid) specialists and advanced tools and not the general user populations.
My point is that, if that is true we do not need to release insecure software. Do the magic before the release and no one gets hurt.
AgentME 20 hours ago [-]
We could rename "cooldown" to "pre-release period".
password4321 1 days ago [-]
The point is to allow the automated scanners a chance to run.
Every security company and their cousin wants to be the one to find the next big dependency malware.
x0x0 1 days ago [-]
The idea that a package can be updated and with a deploy at the right time could be live on your servers in prod 10 minutes later has always been crazy, and the last years have just reinforced that.
People are encouraged by package managers to treat any bit of code someone tosses onto a package manager as equivalent in reliability to the core language and sdk.
nicoburns 17 hours ago [-]
Yep! We ultimately need better tooling for crowd-sourced manual auditing, not just a delay.
ihumanable 1 days ago [-]
Yea, all the new advice around using dependency cooldowns only works if _someone_ is installing these things before you and finding the vulnerabilities.
It seems like the advice right now is to become a freerider while there are still people installing closer to release that will do free work for you finding out there's something nasty in the release.
Once everyone is waiting 2 weeks to install an update, then the value of everyone waiting goes down dramatically.
postalcoder 1 days ago [-]
Using dependency cooldowns is not a free-rider problem. There's a real tradeoff here – ppl are trading their time preference for security.
Just as users are incentivized to avoid malware, researchers and attackers are equally motivated to be the first to discover it.
The concern trolling around widespread dependency cooldowns doesn't make sense. Most people shouldn't be eager to download a release that hasn't made its way through at least some scans.
kbenson 1 days ago [-]
This is how a chunk of people function anyway. There are plenty of people that choose to not install "point zero" release for software of a certain importance, assuming with any major changes there are often bugs that come along with it.
In this case, since the number of cool down days is configurable, even if everyone was using it we would still likely see a somewhat smooth curve for adoption, since not everyone will choose the same delay and the delay time will likely map closely to how people want to habdke risk.
It's all a trade off, just like it's always been. This just makes it simpler to act on what you want your risk/comfort level to be.
raesene9 1 days ago [-]
not really, there are a number of security companies doing analysis of any new packages looking for supply chain attacks, so if you wait a couple of days, till their analysis is complete, you're reducing the risk of hitting a compromised package.
teeray 1 days ago [-]
It basically devolves into a Volunteer’s Dilemma. There’s no incentive here to be the guinea pig, so nobody will want to be.
embedding-shape 1 days ago [-]
> It basically devolves into a Volunteer’s Dilemma. There’s no incentive here to be the guinea pig, so nobody will want to be.
Except there is lots to gain from being the first to write about the new malware on some registry, so companies are actively downloading and inspecting literally every package.
Back in the day (maybe 6-7 years ago?) you could detect this by uploading a new npm package that hit back some endpoint in your control, and it was almost guaranteed that this endpoint got a request within a minute of publishing a new package or update to existing one with users. Nowadays I think none of the scanners actually run the code, mostly static-analysis, and I dunno how often the npm download counter updates per day, probably harder to see in real-time.
teeray 1 days ago [-]
> Except there is lots to gain from being the first to write about the new malware on some registry
Show me the company writing to their customers “we intentionally decided to ship code with potentially novel vulnerabilities. One of those vulnerabilities caused disclosure of your data, but cheer up! We have this cool security blog post about it now.” Meanwhile their competitors freeride and their customers’ data is safe.
scheme271 24 hours ago [-]
I think it's more some security company writing about a vulnerability they discovered in this module or a worm/backdoor and not the company that wrote the software. The security company gets publicity and potentially gets more biz for security consulting.
weaksauce 1 days ago [-]
security researchers not the ones shipping the faulty code.
teeray 1 days ago [-]
We’re not talking about security researchers here:
> there is lots to gain from being the first to write about the new malware on some registry, so *companies* are actively downloading and inspecting literally every package.
(Emphasis mine)
weaksauce 23 hours ago [-]
yeah security researchers at security companies are the ones we are talking about.
john_strinlai 24 hours ago [-]
>We’re not talking about security researchers here:
we are.
"companies" in this context is "security companies" (hence why they are "downloading and inspecting every package", which would not make sense if referring to the people authoring and shipping a single package)
eranation 17 hours ago [-]
Awesome!
Will add it to depsguard.com this weekend.
tancop 1 days ago [-]
[dead]
delichon 1 days ago [-]
> A version whose source does not expose created_at, such as older gem servers, historical entries from before the v2 cutover, or private registries still on the v1 format, is treated as outside the window and stays resolvable.
How is that not an easy exploit to circumvent the cooldown?
werdnapk 1 days ago [-]
Most gems in Ruby/Rails projects come from rubygems, so if they were published long ago, any exploits should have already been found hopefully. Any old gems that would attempt to release a new compromised version would now get a created_at timestamp and the cooldown applies.
Unless you can compromise the gem server to overwrite created_at fields, I don't see any exploits here.
Private gem servers are either already trusted (if they're your own) or already under some scrutiny and extra care already being taken (ideally), but this last case applies to very few projects I'm sure.
tenderlove 23 hours ago [-]
RubyGems.org backfilled older releases with `created_at` fields, so theoretically you could still do the cooldown with very old gems (though I don't know why you would). It's only private / alternative gem servers that may not provide `created_at` fields.
OptionOfT 1 days ago [-]
Can you in your own gem depend on gems from another server? Or does it need to be configured on the client?
If not, and the current defacto standard gem server doesn't accept v1 anymore, we're good I suppose?
doctorpangloss 1 days ago [-]
you have 1.0 installed. you enable 7 day cooldowns. an exploit is discovered in 1.0, and 1.1 is immediately released to fix the exploit. do you sit on 1.0 for 7 days?
trevor-e 1 days ago [-]
it specifically addresses this in the "The escape hatch" section...
k3nx 1 days ago [-]
So, the threat actor now, after making the compromise, just needs to announce that the previous version has a 0-day, and folks need to install the latest version? I love the idea of a cool down, but it can still be thwarted. I would just hope folks that are trying to patch a 0-day take extra caution to vet the new version. I wouldn't be opposed to a --cooldown 0 doing a side by side diff. I may not know what's going on in the code, but a 0-day shouldn't be a ton of new code either.
esafak 1 days ago [-]
Security updates bypass the cooldown.
doctorpangloss 19 hours ago [-]
But what channel decides it is a security update? How do you know? Someone has to notify whom exactly? And what if the adversary says their supply chain attack commit is a security update?
All of this cooldown stuff is so mind bogglingly stupid...
1 days ago [-]
shevy-java 1 days ago [-]
Meanwhile ruby is dropping ranks.
How active is rubygems.org itself? I retired when the 100k download threshold was installed onto developers there; on github I don't have any such restriction pertaining to code I publish and maintain. But even before that restriction, numerous gems were abandoned. I understand that this is a natural cycle anyway, but without an influx of new developers, ruby will fossilize and age out just as perl did before.
None of those "cooldowns" will bring in new developers either. It all seems to be about meta-appeasing companies; this could indirectly help, but I doubt it will help much.
Rendered at 18:23:21 GMT+0000 (Coordinated Universal Time) with Vercel.
So, just like COVID-19 used air travel, modern malware attacks are relying on GitHub+dependabot to speed up the spread.
Even for single page website built using Vue, I would get about 5 updates a week.
For dependabot it's as simple as
cooldown.default-days: 1
There are security researchers (that don't have cooldowns) that usually detect compromises within hours or less, and package managers almost always manage to remove the offending versions in less than 24 hours (usually much less).
So people will 24 hours cooldowns get protected.
Shameless plug: I maintain depsguard.com that tries to simplify cooldowns setup across anything that supports it, in one command (it scans from where you run it, e.g. if you run it from your user folder it will look for any local repos with dependabot / renovate and suggest a change.
Most people stick to default of 0. In fact, I am realizing over time that it is best to make it 7-14 days.
More than I think should be, frankly. More than enough for a sustainable industry.
As for “which firms”: if you Google any of the recent dependency compromises, you’ll see their names. My rough guess is that there are somewhere between 12 and 20 active players in the “supply chain security” space, and they generally compete for mindshare with blogspam. That’s not to say their scanning results aren’t good, though.
I think it remains to be seen whether the economics of this kind of “supply chain” compromise support a stealthier actor profile: the whole reason to compromise an OSS project rather than exploit a single victim’s browser or phone is to smash-and-grab en masse. That’s a fundamentally noisy and delay-sensitive attacker profile.
You need an early release in the "given enough eyeballs all bugs are shallow" world because you need the eyeballs, but if you count on specialists and scanners no general availability release is necessary and hence no cool down.
It is true if everyone waits a cooldown (I don't like this name, but that's another thing) period, then it doesn't solve anything, but as others have pointed out, just because you don't build off the cooldown doesn't mean you and others aren't watching the releases.
Closing the window makes the burden of hiding the exploit higher, which is I think objectively an improvement of security posture.
in the "find the vulnerability before release" scenario, how do I push version foo-3.5? is it my responsibility as the package author to find a security firm and have them audit it before submitting to the repository? should the org owning the repository shunt my package update to a holding pen until it can be vetted, and if so do they have to pay a designated security firm and give them special access to the holding pen, or can anyone reads the incoming packages and help vet them? and if no one has looked at a package in a while do you delay it indefinitely or so you let it time out and be released? that's basically back to the cooldown idea, only implemented on the server rather than the client, and therefore not letting individual clients tune what they believe is a good cooldown time.
the releases will be delayed for the same time period, but you increase the amount of coordination required significantly and reduce user agency.
One idea to get out of this mess is to use cooldowns. But this idea only works if what GP said is true: Vulnerabilities are mostly found by (paid) specialists and advanced tools and not the general user populations.
My point is that, if that is true we do not need to release insecure software. Do the magic before the release and no one gets hurt.
Every security company and their cousin wants to be the one to find the next big dependency malware.
People are encouraged by package managers to treat any bit of code someone tosses onto a package manager as equivalent in reliability to the core language and sdk.
It seems like the advice right now is to become a freerider while there are still people installing closer to release that will do free work for you finding out there's something nasty in the release.
Once everyone is waiting 2 weeks to install an update, then the value of everyone waiting goes down dramatically.
Just as users are incentivized to avoid malware, researchers and attackers are equally motivated to be the first to discover it.
The concern trolling around widespread dependency cooldowns doesn't make sense. Most people shouldn't be eager to download a release that hasn't made its way through at least some scans.
In this case, since the number of cool down days is configurable, even if everyone was using it we would still likely see a somewhat smooth curve for adoption, since not everyone will choose the same delay and the delay time will likely map closely to how people want to habdke risk.
It's all a trade off, just like it's always been. This just makes it simpler to act on what you want your risk/comfort level to be.
Except there is lots to gain from being the first to write about the new malware on some registry, so companies are actively downloading and inspecting literally every package.
Back in the day (maybe 6-7 years ago?) you could detect this by uploading a new npm package that hit back some endpoint in your control, and it was almost guaranteed that this endpoint got a request within a minute of publishing a new package or update to existing one with users. Nowadays I think none of the scanners actually run the code, mostly static-analysis, and I dunno how often the npm download counter updates per day, probably harder to see in real-time.
Show me the company writing to their customers “we intentionally decided to ship code with potentially novel vulnerabilities. One of those vulnerabilities caused disclosure of your data, but cheer up! We have this cool security blog post about it now.” Meanwhile their competitors freeride and their customers’ data is safe.
> there is lots to gain from being the first to write about the new malware on some registry, so *companies* are actively downloading and inspecting literally every package.
(Emphasis mine)
we are.
"companies" in this context is "security companies" (hence why they are "downloading and inspecting every package", which would not make sense if referring to the people authoring and shipping a single package)
How is that not an easy exploit to circumvent the cooldown?
Unless you can compromise the gem server to overwrite created_at fields, I don't see any exploits here.
Private gem servers are either already trusted (if they're your own) or already under some scrutiny and extra care already being taken (ideally), but this last case applies to very few projects I'm sure.
If not, and the current defacto standard gem server doesn't accept v1 anymore, we're good I suppose?
All of this cooldown stuff is so mind bogglingly stupid...
How active is rubygems.org itself? I retired when the 100k download threshold was installed onto developers there; on github I don't have any such restriction pertaining to code I publish and maintain. But even before that restriction, numerous gems were abandoned. I understand that this is a natural cycle anyway, but without an influx of new developers, ruby will fossilize and age out just as perl did before.
None of those "cooldowns" will bring in new developers either. It all seems to be about meta-appeasing companies; this could indirectly help, but I doubt it will help much.