> In my experience the kinds of people who run this exact combination of circumstances also tend to be the kind of people that have a wide variance in the level of kindness they display to the authors of open source programs that happen to be in their way.
Love this. There’s been past discussion on HN re how OSS maintainers are treated, and this is such a wry sentence. I really appreciate the tone / attitude to the problem.
skrebbel 1 days ago [-]
Can you translate the American for me? “Wide variance”, does that they’re being assholes?
st_goliath 1 days ago [-]
> Can you translate the American for me?
I'd say "wide variance in the level of kindness" is a rather British way to describe some of the opinions that have been voiced.
cowsandmilk 23 hours ago [-]
100%, author lives in Canada and the phrasing has the British heritage of Canadian English.
graemep 20 hours ago [-]
Sounds rather like a statistician's way of saying it.
afavour 1 days ago [-]
Well a little translation would be that some are very kind, some are total assholes, and everything in between. But your reading is probably more accurate.
strken 1 days ago [-]
I think the literal interpretation is correct here. It's a way of saying "you, dear reader who runs this combination of things, might be very kind, but some of your compatriots will be unusually unkind and will turn up in my issue tracker and on my socials to complain".
1 days ago [-]
skrebbel 1 days ago [-]
Thanks!
joquarky 15 hours ago [-]
Same goes for a job writing closed source. This whole industry has gone off a cliff.
doctor_radium 1 days ago [-]
I have every copy of Firefox here configured with webassembly disabled...because I don't tend to do what Webassembly was designed for, i.e. online games, video/audio editing, emulation, etc. [1] and because I dislike things running in the background without my knowledge.
So this is going to be interesting.
My plea (in a small voice) to the OP: just as many sites still do for JavaScript, please add a "This captcha required Webassembly to continue" message to your code when Webassembly is disabled. And, incidentally, for smaller platforms/browsers that don't offer a Webassembly engine at all.
The second half of this post chronicles the author's Herculean efforts to make everything work well in the cases you describe.
> One of the big things that blocked this shipping for so long was not having an escape hatch of some kind to allow clients that disable WebAssembly by policy to get through the gate. [...] This combination of factors means that there would need to be another implementation of the proof of work code in JavaScript that would actually execute the number crunching.
doctor_radium 1 days ago [-]
Thanks. You're right; it was a very thorough article and I trailed off about halfway through. I still say there are many improper uses of Webassembly out there, but I credit the developer for a highly thoughtful approach to the project. I really wonder if Cloudflare et al. does as much due diligence?
swiftcoder 1 days ago [-]
> because I dislike things running in the background without my knowledge
Point of order: WASM doesn't let things run in the background. To do that you need WebWorkers (a javascript feature, not WASM-specific).
zamadatix 16 hours ago [-]
Two different kinds of "background". GP is talking about what indications they get a page is running arbitrary code as they load it, (some of) web workers is about persisting running that code after it's loaded. It can be in the "background" for both or neither, depending on the context, and both meanings are valid.
swiftcoder 14 hours ago [-]
> GP is talking about what indications they get a page is running arbitrary code as they load it
I mean, you get the same indicators as when the page runs arbitrary Javascript on load? WASM assembly may be (slightly) harder to read than typescript compiled to JS, but the source is all there, it calls the same APIs, etc.
zamadatix 12 hours ago [-]
They are not saying:
js on any site -> okay, I enable permanently
wasm on any site -> not okay, will never enable
But rather:
js disabled -> I often get a fallback warning message letting me know a site needs it to work
wasm disabled -> I'd like a similar kind of message
So they don't have to guess at the cause(s). I.e. they don't mind Anubis moved to WASM over JS in itself, they mind there is no fallback warning message letting them know that's the missing requirement. In general, I agree with the problem trend. Most users get pushed along auto-update tracks these days and so fallback behavior has often fallen out of mind with the newer technologies even though it's so easy to implement. That's almost expected with "normal" pages and the lack of time to put into them but when you push something which will affect many sites it's probably worth the reminder for a fallback message.
swiftcoder 2 hours ago [-]
Right, but the base assumption here is unfounded. Disabling WASM isn't in any way equivalent to disabling Javascript - you are just kneecapping your own browser performance, rather than closing off a large attack surface.
noduerme 1 days ago [-]
It's amusing (although disheartening to me as someone who builds online games) that wasm is now being treated like a flash plugin by sysadmins. You're not entirely wrong, but it's just funny to me that we took a full ten years to get back to roughly the same place. Now with Ruffle running Flash in wasm, no less! We build shit that doesn't work in the DOM, search engines hate it, walled gardens try to block it, admins fear it, they shut it down and the whole thing starts over.
xena 1 days ago [-]
I don't mean to be rude, but please search "wasm2js" in the article.
DrammBA 21 hours ago [-]
Some times it's okay to mean it.
Plont 17 hours ago [-]
This is not one of those times.
tiagod 19 hours ago [-]
Isn't the WASM attack surface absolute minuscule when compared to JS? Do you also disable web workers?
doctor_radium 8 hours ago [-]
I haven't disabled web workers, but am playing with the concept. T-Mobile's payment portal screams and then bans my IP (something like that) if try. I have disabled service workers globally, and so far no fallout.
nosioptar 15 hours ago [-]
I disable workers.
I forgot to do so on my phone. I noticed Firefox had slowed to a friggeb crawl when opening pages. Disabling web workers fixed that.
jonkoops 1 days ago [-]
I really don't see why you'd disable WebAssembly and allow JavaScript instead. Expecting developers to make a fallback for a baseline-supported feature for any reason but backwards compatibility with runtimes that do not so support WASM is extremely silly.
doctor_radium 7 hours ago [-]
I also have JS disabled in Firefox on my phone. It's on by default on my laptop, as I apparently visit different sites when my butt is planted in a chair.
My wish is actually for the days of NPAPI where Webassembly would be just another plugin (like Flash) and browsers clung to mainly HTML, JS, and CSS. Then developers wouldn't be quite so keen to rely on it and the web would be much less bloated. Yes, those days are long gone... As a tradeoff, I'd settle for a simple toolbar icon that lit up when Webassembly is running, and/or a high level UI (just like Location, Camera, and Microphone) so I can enable it for just those sites that need it. I tried selling Mozilla on the idea once but they apparently weren't interested.
Baselines aren't entirely universal. I remember the days when the Mozilla (or early Firefox) browser was the only viable web client for Linux, but some sites still opted to run behind ActiveX. Today I don't think you'll find a Haiku or Amiga browser with Webassembly support, nor Firefox for Linux on IBM Power motherboards. It's easy to forget that just because most any 'nix or BSD OS has a plethora of choices, that others aren't still dealing with the "ActiveX" situation and feeling like everybody's out to get them.
well_ackshually 1 days ago [-]
"I disable things and expect others to do things for me, in their own time, without being paid, to bring solutions to my paranoia"
You do see how selfish it is, right? While the author did introduce such a solution, it's such a massive waste of time.
xenafan1 1 days ago [-]
[flagged]
gavinsyancey 1 days ago [-]
> You should really be running the most popular browser
This amounts to giving Google effectively complete control over the web. I'm not sure that's a good idea. IMO it's important that Firefox and Safari remain popular enough that webdevs are forced to support them.
I'm somewhat sympathetic to "If you want to 'customize' your web experience, you shouldn't be surprised when things don't work right." in terms of turning off commonly-used features, but the request isn't to make it work right. OP just wants a helpful error message.
1 days ago [-]
adrianN 1 days ago [-]
Customizing the experience is what the web is all about. The browser is a user agent not an advertising platform for the corporate overlords.
kccqzy 2 days ago [-]
Hats off to Xe for spending so much time on backwards compatibility, especially the tidbit about targeting Chrome 66.
I have a Mac from 2014 running Yosemite that I occasionally use to test for backwards compatibility in my own frontend code (for fun!). But IMO the best way to ensure compatibility is to use period-correct toolchains or toolchains where the pace of change is slower, like ClojureScript.
nxobject 1 days ago [-]
In the non-webdev realm, it's gobsmacking that you can put "FROM debian/eol:buzz" in a Dockerfile, and it will do what it says...
yunaflox 20 hours ago [-]
Just use Momiji, unless you care about Chrome so much. In that case, Chromium Legacy might work.
adrian17 1 days ago [-]
> something I was doing with my "strict MVP" build of Anubis' WASM wasn't in fact sticking to just the MVP features of WebAssembly
Fun fact, this _used_ to be the case - wasm32-unknown-unknown had extra non-mvp features added later, which in my eyes has been a breaking change on stable. You're not the first person to have been bitten by this, there's [1] and a similar story in Ruffle [2].
Weirdly, the response from the rustc side (IIRC we also talked with one of the devs of wasm component on discord a bit around 2024) has contained a lot of (paraphrasing) "in retrospect we think wasm32-unknown-unknown target was designed badly*, so we'd prefer if you either dealt with it or switched to another wasm target <with its different set of downsides> than us fixing it". (*I acknowledge that the way std was implemented on that target was unusual and "not clean", but it works _really well_ for a lot of people.)
> I looked into the process involved for rebuilding the standard library twice: once with only MVP wasm features enabled and once with an "all yes config" like usual. Based on some research I did this seemed like a massive pain.
That's what Ruffle does [3], it's a bit unfortunate but wouldn't call it a massive pain.
> Weirdly, the response from the rustc side (IIRC we also talked with one of the devs of wasm component on discord a bit) has contained a lot of (paraphrasing) "in retrospect we think wasm32-unknown-unknown target was designed badly, so we'd prefer if you either dealt with it or switched to another wasm target <with its different set of downsides> than us fixing it". (I acknowledge that the way std was implemented on that target was unusual and "not clean", but it works _really well_ for a lot of people.)
That's about right. Specifically, wasm32-unknown-unknown is serving multiple targets, which aren't very compatible, and it implements std by panicking in all the bits that can't work.
One target is "you truly have absolutely nothing, and don't really have much of `std` at all". Another target is "you're in a browser, and can do anything a typical browser can provide" (ideally targeting the future where wasm has full browser APIs without going through javascript). Another target is "hosted environment that isn't a browser, and which capabilities you have depends on the environment". Those are three very different things.
I would argue that we need at least two new targets, possibly three:
- `wasm32-none` or similar (by analogy with `x86_64-unknown-none`), a target where you don't have anything except for what you bring with you. The equivalent of writing kernel code.
- `wasm32-wasi-hosted` (or pick another name), where you the hoster of the wasm can supply whatever capabilities you have available.
- `wasm32-wasi-browser`.
It's possible that `none` could be a special case of `hosted`, where the services you provide are "none".
Aachen 2 days ago [-]
Is there a place where I can try out if my browser is compatible? Easier to find out now than when I'm trying to get work done and a million websites now have it deployed
On https://wasm-feature-detect.surma.technology it shows that I don't have 3 of all these features but I'm not sure if Anubis needs any of them to not kick me back to the pure JS solution
Which would apparently be bad because
> The WebAssembly that's shipped with this flow is ridiculously performant. This may mean you need to adjust the difficulty [upward, to avoid that bots solve it trivially when they support wasm, I assume this means]
combined with
> The wasm2js flow doesn't currently have a way to update the progress bar [so you have no clue about remaining time]
(wouldn't be the first time that I gave up on a page because it was stuck on 0 hashes per second)
xena 21 hours ago [-]
If you don't disable browser features for fun and/or sport, Chrome as of version 120 or later (or another browser of that vintage) will use the optimized flow.
It would be cool if the POW could be done ahead of time. That way I don't get stuck waiting while I'm working. Some type of credit/tokens that my browser could then spend.
That just creates another problem: if you're taking any measures to reduce tracking (ie. clearing cookies on shutdown or using temporary containers), this won't work. If anubis was being deployed on a site that a user visits often (eg. HN), the user might be convinced to whitelist it, but most anubis deployments are on random blogs or fediverse instances that I might not visit again in months. I'm certainly am not going to whitelist those sites, nor am I going to enable cookies wholesale just to avoid solving challenges.
xena 1 days ago [-]
Look, if you're going out of your way to break expected behaviour on websites you shouldn't be surprised when people code to the most common denominator and then you have weird subtle breakage as a result.
kstrauser 1 days ago [-]
Gotta agree. Turning tracking cookies off? Sure! Turning session cookies off and then complaining that the server doesn’t remember you? Uhhh…
And while I’m sympathetic to the idea of not wanting to run JS, to a first approximation modern browsers are JS engines that have graphical displays. How things should be vs how they are is a classic is/ought problem. The world took a vote on what a browser’s meant for and we lost. Fighting it today is rough; tomorrow, futile.
gruez 1 days ago [-]
>Gotta agree. Turning tracking cookies off? Sure! Turning session cookies off and then complaining that the server doesn’t remember you? Uhhh…
No, because there are technologies that don't have this issue, eg. privacy pass.
perching_aix 1 days ago [-]
There's no mechanistic difference between the two types of cookies, let's not pretend this is an actual config choice that people are making.
Whenever I hit Anubis, I simply go "keep your secrets then" and take my leave. The vast majority of the sites posted to HN (that catch my fancy) work fine or fine enough, and are better for it.
If I really, really desperately want to check something out, I can always just turn stuff back on. Turns out, I rarely do.
Shoutout to the particularly dogshit few that don't just require cookies and JS, but even third party JS. For reading a blogpost or a message thread!
bmacho 22 hours ago [-]
> There's no mechanistic difference between the two types of cookies, let's not pretend this is an actual config choice that people are making.
They probably meant 3rd party cookies. It's common to block those. 1st party cookies not so much.
nosioptar 15 hours ago [-]
The Anubis anime girl is an instant tab close for me, especially while in public. I dont like having to explain to people that I'm not one of those creepy anime guys.
kstrauser 6 hours ago [-]
That’s the most insecure thing I’ve heard in a while. What other things randomly flashing across your screen disturb you? Do you imagine your coworkers keeping count of which ads are displayed in your browsers? Do you imagine that they don’t also see the Anubis loading screens?
guydoom 9 hours ago [-]
It isn't even anime style, it gives me more western cartoonish vibes.
gkbrk 1 days ago [-]
Clearing cookies when all you want to do is read static content is not "breaking expected behaviour on websites".
anuthis 1 days ago [-]
[flagged]
lxgr 2 days ago [-]
Or something that shadier websites could calculate ahead of time in your browser, then spend on your behalf.
I feel like Anubis is ironically speedrunning a lot of discoveries the crypto folks have already made several years ago...
Terr_ 1 days ago [-]
What if the work output was bound to a particular source IP? That'd stop most "theft" at the expense of discards when your device switches networks.
Happy Eyeballs and switching to/from mobile are definitely hard problems to deal with in a privacy-friendly way, but I don't think IPv6 privacy extensions should really matter here. Just treat a /64 the same way you'd treat a /32 in IPv6-land. It may represent multiple users, but they're all sharing the same internet connection which is as much as the global internet really wants to tell you. Whatever's on the user-controlled side of the address is of course untrustworthy anyways.
I wonder if you could convince the Anubis author to implement x402 payments to bypass the PoW
JsonCameron 1 days ago [-]
This could be done at the web server level, not requiring changes to Anubis
packetlost 16 hours ago [-]
It could be, but if it's a few lines of config in Anubis it would be easier and more likely to actually be done.
apoorvkhanna88 1 days ago [-]
[flagged]
dist-epoch 2 days ago [-]
Then we could implement an exchange, so that if you generate too much Anubis POW, you could exchange it with others.
We could link it to a site, you generate for HN, I for Reddit, but it so happens that you visited Reddit more and I HN, so we depleted our Anubis POW, so we could exchange some Reddit Anubis with some HN Anubis.
ssl-3 2 days ago [-]
And then, we can inextricably link it to a new cryptocurrency -- let's call it BotCoin -- and make Sam Altman pay for it!
kelvinjps10 2 days ago [-]
Basically reinventing crypto one step at the time.
omoikane 1 days ago [-]
> This means that adding one (1) to the difficulty of a challenge makes it 1024 (one thousand twenty-four) times as hard to solve in the worst case.
I don't understand the units here, wouldn't a bit versus nibble difference make the multiplier 16, instead of 1024?
xena 1 days ago [-]
Oops, there's your proof I didn't write this with AI! Fixing, sorry.
andai 1 days ago [-]
Tangential but all the pages I see protected by Anubis are just static HTML. But bot traffic is only harmful on dynamic pages right?
JodieBenitez 1 days ago [-]
I'm not a fan of Anubis for various reasons but the idea that bot traffic in only harmful with dynamic pages must die. CPU (yes, even to serve static pages) is not free, bandwidth is definitely not free. There's an idea that serving a static page to a bot has a marginal cost tending to zero, but it's never really zero and serving them by millions definitely has a cost.
Also, while some pages may look like static HTML pages, they may be generated on the fly by an expensive and/or slow backend, which adds to the cost. I happen to maintain servers for academics and some content management systems are slow and have an expensive CPU cost. While it's OK for the low number of humans interested in the subjects they deal with, it's definitely not fit for massive bot scrapping. And before you ask, no, it's not always practical to have cache upfront or to pre-generate all pages.
GuB-42 23 hours ago [-]
Anubis is not free either, it is a matter of how much it costs to run Anubis vs to let bots in.
When I see LKML using Anubis when the pages it serves are tens of kB, all presumably static, I wonder if they do it out of spite rather than to protect their servers.
As a user/reader/viewer I absolutely hate Anubis and usually turn around when I see it pop up (at least on my phone where it takes ages to compute), but with stats like that, I get why a site operator would resort to using it.
alexey-salmin 8 hours ago [-]
I think the kernel.org post proves the parent point rather than contradicts it.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
14 CPU cores total for running a website like kernel.org is laughable. This is not worth burning cycles in Anubis on client's devices, this is not worth the time of the engineer who worked on it. Provisioning more hardware would have been literally better for everyone.
Gracana 7 hours ago [-]
I can’t say I really disagree, and as a visitor of the site that’s the solution I would prefer.
glub 20 hours ago [-]
> But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.
This drives me crazy with so-called SOTA LLMs that have "achieved AGI".
Fable, Sol, Astra, will start by trying to reverse engineer a binary to figure out how something works when software is open source and one search query away.
You let them know it's open source, and they will start using github API instead of just cloning and grepping.
arcfour 1 days ago [-]
There are numerous services that will let you host static pages for free or nearly free. There are also numerous services that sit in front of your website that can block bots and reduce load on your origin server, many of which are also free, or very low cost relative to the service they provide.
The situation you are in is far less dire sounding when you consider that you have these options available to you.
JodieBenitez 1 days ago [-]
Except that I don't have these options per employer policies.
arcfour 19 hours ago [-]
So your employer is having a problem, and prevents you from using any of the available options to solve it? And you have asked them about all of them/told them about the problem? They don't like saving money?
Well, sounds like it's not your problem then.
account42 18 hours ago [-]
> Also, while some pages may look like static HTML pages, they may be generated on the fly by an expensive and/or slow backend
Yes and that should be fixed before you subject real users to resource-wasting scripts.
JodieBenitez 17 hours ago [-]
It can't always be fixed. Also, I don't use Anubis.
bryan_w 4 hours ago [-]
It's fixed enough for how much they want it to be fixed (I.e. how much they are being paid to fix it)
xena 14 hours ago [-]
Please submit a fix to cgit then.
matkoniecz 15 hours ago [-]
Sadly, I do not control resource-wasting scripts running crawlers.
And to answer your intended demand: this is in some cases impossible or unreasonable. And things were working fine before LLM DDoS.
andai 20 hours ago [-]
May I see it?
jeroenhd 1 days ago [-]
Shitty scrapers hitting images over and over again is also a problem. Not just /pic/1.webp - /pic/2.webp, but in practice I've seen three AI scrapers that would request things like /pic/1.webp?version=webp&foo=bar and other random query parameters that none of my domains even use. It's just stuck brute-forcing files or something, sucking up loads of bandwidth downloading the same image for no reason.
It also depends on if you want your content to be scraped. Any bot ignoring robots.txt should be considered potentially malicious and worth considering such mechanisms for.
account42 18 hours ago [-]
> Shitty scrapers hitting images over and over again is also a problem. Not just /pic/1.webp - /pic/2.webp, but in practice I've seen three AI scrapers that would request things like /pic/1.webp?version=webp&foo=bar and other random query parameters that none of my domains even use
That's a trivially solved problem - have the web server refuse (or redirect if you're feeling nice) requests with unexpected query parameters. No reason to make things worse for real users before optimizing your site.
And even if there are actual parametric resources you want to serve, apply limits only to them and let users visit the rest without unnecessary bot checks.
jeroenhd 15 hours ago [-]
Sure, there are loads of easy solutions to this problem. Putting everything behind Cloudflare is the most common one, Anubis and its competitors are another. I could also block all data centers, countries with loads of CGNAT + botnets like Brazil, China, and probably India, and refuse any user agent not in the top 5 modern user agents with a little TLS fingerprinting to make sure.
I could probably also analyze all the paths across my servers and write some bespoke Nginx config files to filter out all requests with illegal queries, but that's harder, more fragile, and just leaves the servers open for the next time a scraper finds a new way to screw me over.
The bots I'm fighting have gone for the nuclear option, relying on botnets and spoofing to bypass basic filtering, so I don't see why I shouldn't do the same.
zamalek 16 hours ago [-]
It protects the content from being hoovered up into an LLM without credit or attribution. Anti-DOS is only half the story.
trickypr 1 days ago [-]
The anubis docs website is used as an example of how anubis works.
Also, if you are not using a big CDN (e.g. Cloudflare), you still need to pay for the traffic (even if you don't pay as much for the CPU). This is doubly true for smaller CDNs like Bunny.
xena 1 days ago [-]
Not only is it an example, it specifically is one of the first things I use to test experimental features so I get data back instantly.
account42 18 hours ago [-]
You also need to pay for the traffic it takes to server you bot check scripts. And outside of big cloud companies, traffic is actually dirt cheap.
Nnnes 1 days ago [-]
Others have answered your question. A lot of the sites listed here https://anubis.techaro.lol/docs/user/known-instances/ have dynamic content; Git web interfaces in particular (Codeberg, the Linux kernel, FFMPEG, and more are on the list) are vulnerable to poorly or maliciously configured scrapers.
andai 20 hours ago [-]
Yeah it makes sense for dynamic content. But so far I have only seen it on blogs, which could have just been a html file.
account42 18 hours ago [-]
Even for git hosts, there is no reason to add bot checks to e.g. the repository root or other common URLs that random real users land on.
tyre 1 days ago [-]
some people don't want to be scrapped at all, and most bots ignore `robots.txt`
kiicia 1 days ago [-]
no, absolutely not, any malicious or dishonest bot traffic is harmful, generating unnecessary transfer and literally stealing data
andai 19 hours ago [-]
> malicious or dishonest bot traffic is harmful
I only hear bad things about bots lately, are there any benevolent bots?
I've done a tiny bit of scraping myself (usually to archive old websites before they die), and I've gotten banned a few times even though I wasn't hammering anything.
Looks like it's getting harder and harder to do scraping for legitimate purposes. Although I suppose legitimate is subjective :) Everything becomes valuable when it's old enough...
keeda 7 hours ago [-]
I completely understand this change (esp. with regards to the change in type of problem) but I must appreciate the irony that this is essentially making it more efficent -- i.e. reducing costs -- to do a proof of work task, which aims to increase costs ;-)
miraline 1 days ago [-]
The interesting part is that bots mitigation is actually an economic problem. You don't need to make scraping impossible, you only need to just make it expensive enough so that abusive traffic stops being worth it.
mitxela 1 days ago [-]
Scrapers scrape all sorts of nonsense, notably every possible git diff from git hosts. They clearly aren't concerned with rationality.
XorNot 1 days ago [-]
Scrapers scrape every link they can find.
The git host example is probably the one which Anubis is the dumbest defense for: the main reason to have those links is for easy machine interaction. So that's dead when you implement it.
So why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.
mitxela 1 days ago [-]
The main reason to have those links is for people to click on them. No sensibly programmed machine is scraping diffs from a git web front end.
novafunc 22 hours ago [-]
That's exactly what's happening to the Linux git repository. The server constantly has mutltiple CPU cores dedicated to rendering git of various old, dead branches in html for scrapers.
cyphar 1 days ago [-]
Not to mention it is the least efficient representation of git commits. It seems a lot of people are unaware that git generates the diff representation you see with "git show" on the fly -- internally, commits are Merkle trees representing a snapshot of the repo file contents.
As you said, if you actually want to interact with a remote git repo you'd do so with packfiles not diffs.
utopiah 1 days ago [-]
> why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.
I document my code, in issue and elsewhere on the Web, by referring to specific commits so having a URL for each is useful to me at least.
worldsavior 21 hours ago [-]
It's already expensive, though they still profit. Let's see if this makes it more expensive, since RAM/storage is much harder to come by these days.
zamalek 16 hours ago [-]
> It also means that the "hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
Would there be any risk in making a dynamically built algorithm (i.e. the risk that we don't find N zero prefix bits)? Chain a set of transforms and build a unique WASM binary (including optimizations to deter pattern matching); possibly one a day or hour or something. That way you'd need to have Claude vibeslop the CUDA implementation at that cadence, per website.
xena 16 hours ago [-]
This is exactly what I've been working on. For obvious reasons around it not being done yet, I don't have anything else to say.
zamalek 13 hours ago [-]
That's awesome!
raincole 2 days ago [-]
It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?
solid_fuel 2 days ago [-]
The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
Aachen 2 days ago [-]
> requiring 1 second of compute before allowing a page load
After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves
Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing
jdlshore 2 days ago [-]
The issue is that scrapers are rotating their IP and essentially performing a DDoS attack. Anubis is part of a defense-in-depth solution. If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.
Aachen 2 hours ago [-]
I don't know what you mean by traditional anti-abuse (typing over two words from an image?), but switching IP addresses rapidly, while staying recognisable as one entity because of the unique cookie, is not normal user behavior so this would be trivial to filter out for something like Anubis and require another solve. Maybe that's what you meant though?
gruez 2 days ago [-]
>If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.
Scrapers can't change their strategy? It's not hard to think of mitigations: keep on using a session until you're banned, and failing that rotate your IP. Nowadays you don't even need to think. You can probably tell a LLM "write me a scraper that has adaptive anti-anti-bot mechanism".
embedding-shape 1 days ago [-]
> Scrapers can't change their strategy?
Yes, but you normally don't try to fight all strategies up front, you add stuff on a need-to basis. Currently, lots of people are in the stage of "need to get this thing using bunch of IPs to stop hammering my servers", apparently Anubis helps them with that. If that stops working, other tactics are tried, and so it goes on for the life of your service. Hasn't stopped yet, doesn't seem to be stopped soon either.
gruez 1 days ago [-]
That argument also works against anubis. Why bother forcing clients to solve a PoW challenge, when you're fighting a bunch of unsophisticated attackers that will be deterred by a javascript challenge that requires them to solve `result = x + y`?
Turns out that in practice there's some bots that implement support for this but not JavaScript.
embedding-shape 1 days ago [-]
Point is, you get problems because of something like this, you try either approach. And when that stops working (because eventually it will), you add something else. Some work better against some actors, but also feels like it never ends, so slowly you add more things. I don't think Anubis is uniquely able to stop that particular type of annoyance, it's just one of the tools you can see if it helps. If you want to try something else before, makes a lot of sense to do so.
well_ackshually 1 days ago [-]
You're not fighting unsophisticated attackers. 50% of them are, maybe. The rest? They're actively hostile, detecting blocking attempts, moving on to botnets, DDoSing you and trying everything they can to bypass you.
People have been sent to jail for participating in LOIC attacks which were less destructive that what these bots do on the daily.
mitxela 1 days ago [-]
[flagged]
myng111 1 days ago [-]
Not only is this not true, you are misgendering Xe. They use they/them pronouns.
bulder 1 days ago [-]
Plenty of these abusive scrapers are utilizing retail residential proxies, which will be applying forced rotations to avoid "burning" their compromised and or otherwise surreptitiously utilized IP address.
I would disagree with the Anubis effectiveness falling off, but the llm bros are throwing life itself into training to defeat any “perceived” challenge.
mitxela 1 days ago [-]
If you change your IP address, it invalidates the cookie. If you don't change your IP address, you can be blocked by IP address.
drum55 2 days ago [-]
It’s not seconds though, it’s in the order of tens of milliseconds of work, it’s faster to complete the challenge than most http requests will take.
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
bombela 2 days ago [-]
Yep, on my aging phone it takes quite w noticable amount of time to go through. But on my more recent laptop wnd workstations, it's a quick flash of a manga lolita.
xenafan1 1 days ago [-]
[flagged]
Jtariiiii 2 days ago [-]
>It’s not stopping anyone.
You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
drum55 1 days ago [-]
That’s because its weird and mostly irrelevant, it’s nothing to do with the proof of work being any sort of economic impediment. It takes Claude a few minutes to write a near instant solver, that’s the maximum proof of work cost this adds.
yborg 1 days ago [-]
This strain of nihilism about anti-bot efforts baffles me. Cloudflare built a gigantic business on stopping DDoS attacks. There will never be good faith self-restraint by AI bros, there is too much money at stake and no regulation, doing nothing means your only alternative is disconnect from the Internet. Anubis introduces enough friction that some percentage of the lunacy is throttled and possibly give your server a fighting chance.
gruez 2 days ago [-]
>There are plenty of people demonstrating that Anubis reduces bot traffic.
"Anubis reduces bot traffic" and "[the proof of work] is not stopping anyone" isn't contradictory, especially if anubis isn't being benchmarked against non-POW alternatives like IP/ASN bans, or tls fingerprinting.
Jtariiiii 2 days ago [-]
"it's not stopping anyone" and "it is stopping some people" are mutually exclusive statements.
gruez 2 days ago [-]
You're selectively quoting the comment. It's clearly talking about the PoW aspect not being a serious impediment.
xboxnolifes 1 days ago [-]
Who cares if its PoW or something else? Does it matter in any meaningful way?
account42 18 hours ago [-]
It matters if the PoW component is negatively impacting real users, which it is.
wiredfool 21 hours ago [-]
IP blocks don't work, because they're using proxy networks so that you see an ip address 1 or 2 times within 10 minutes. They have effectively infinite ip addresses. (actually, looking at my data from today, I think this relationship holds over ~3 hours, where we're seeing ip address cardinality at about 1/2 of the hits.)
* Sometimes there's a pattern to the country. Oftentimes, not.
* User-agent, rotated between common, valid, current web browsers.
* Other headers, sec-*, accept, etc, generally valid and rotating.
* Bots will load the site to saturation in a denial of wallet attack.
The only thing that's specific is:
* urls have a pattern.
* it's obviously invalid traffic.
(non-bot traffic on my sites does not go from 0 to 200r/sec on the search interface in seconds. It does not go away that fast either)
gruez 2 days ago [-]
>The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages.
There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.
account42 18 hours ago [-]
Yes, at the end of the day this is DRM and as always it affects legitimate users more than it does those it is meant to block.
fc417fc802 2 days ago [-]
The assumption is that there will always be some population distribution in terms of effort expended as well as in terms of problems caused. At present it's the lowest effort actors that are causing the highest number of problems. So an 80% (or even just a 20%) solution largely solves the practical problem faced by site operators.
If against all odds it turns out that skiddies (or agents or AI billionaires or what have you) are willing to burn inordinate amounts of RAM and CPU just to scrape autogenerated cgit pages that are of approximately zero value to them to begin with then I guess we'll just have to cross that bridge when we come to it.
XorNot 1 days ago [-]
Okay but it's like no RAM at all. Thats the problem: you've got stupid bots causing a bunch of issues - these can be stopped.
And then regular users: who now can't really browse your site.
And then smart bots - anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down.
The solution basically makes the favored customer well funded AI scrapers, not users.
jeroenhd 1 days ago [-]
Regular users store the cookie and do the challenge once. Bots do it over and over again throwing out the cookies so they're not rate-limited.
Extreme configurations with high difficulty do make for a frustrating experience on very old devices, but in practice it rarely gets in the way.
Only the tail end of what one could consider normal users (those running custom browsers or having JS disabled or nonfunctional) run into issues here, but they'll have these issues with any other bot-blocking mechanism that still offers some protection these days.
Apple's "ask your vendor for tokens to exchange for access" web standard gets around this, assuming those bespoke and Javascript-less browsers implement PATs, but that standard seems to be completely dead in the water outside of Safari.
fc417fc802 1 days ago [-]
> And then regular users: who now can't really browse your site.
That's entirely up to how the site operator configures the PoW challenge.
> anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down
Do you have any evidence that it is the large well funded AI companies DoSing sites by (for example) scraping absurd numbers of permutations of cgit url parameters?
Even if it is (which I doubt), if all this accomplishes is getting them to spend 10 minutes optimizing their bots to not do that it will be a major win.
1 days ago [-]
pelcg 2 days ago [-]
[dead]
noobis_4433 3 hours ago [-]
Getting "double_spend" error while trying to access the blog. Anyone else?
arjie 2 days ago [-]
Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.
mitxela 1 days ago [-]
In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.
arjie 1 days ago [-]
That is true, but if you have Anubis you presumably do intend to `Disallow` all bots.
9cb14c1ec0 22 hours ago [-]
Not necessarily. You might want to rate limit bots instead of disallowing them.
pinocchioruns 21 hours ago [-]
[flagged]
xena 2 days ago [-]
One of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.
and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.
2 days ago [-]
dspillett 1 days ago [-]
> It also means that the "hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
I assume that the previous challenges will still be available for circumstances where WASM is not available (unless we are just working on the basis that it is available if JS is?) or perhaps as a fallback. TFA explicitly mentions “smart” TVs, and I'm sure there are plenty of those out there that run browsers old enough to not know what WASM is. TBH these would not be a priority for me, but they obviously are for this project given they (well, TVs generally, not specifically elderly ones) get specific mention in this write-up.
If so, what is to stop scrapers from just forcing the fallback POW option instead of trying to run the new memory-hard algorithm at all?
Lyrex 1 days ago [-]
the answer to this is in the post: where wasm is not available, he transpiles the wasm binary back to Javascript
dspillett 1 days ago [-]
Ah I must have skipped by that bit. And presumably the memory-hard algorithm is CPU-easy enough that this won't be a problem. That leaves me wondering "why not just use a JS memory-hard routine in the first place, but maybe I've shouted over that too. I'll give TFA a reread later.
trickypr 1 days ago [-]
I think it is still compute-heavy if not compute-hard, and stuff like WASM SIMD makes it less compute heavy (not really available in JS). And also, for memory bound problems, you probably don't want the JS GC needing to clean up afterwards when WASM gives you that control.
xena 1 days ago [-]
I assume that SIMD acceleration is usable with WebCrypto, but I have no evidence either way. It's also not part of WebAssembly so there's that to consider too lol
tyre 1 days ago [-]
tangential fyi: the author is not a he
tefkah 2 days ago [-]
thanks for trying to make the web suck a little less these days xena!
evmar 1 days ago [-]
I think the Rust feature you’re looking for regarding recompiling the standard library is called “build-std”, that should be enough for you to search for it. (For similar reasons you also need that flag if you are trying to use Rust to build multithreaded wasm binaries, so it might come up for you!)
jezek2 1 days ago [-]
Great article, WebAssembly is such a good technology when you have use case for it.
I've just finished creating a WASM 1.0 implementation (simple JIT on x86 and x86_64) for plugin usage and I'm looking on how to support more languages beyond C (for which I have "micro-libc" to create really small binaries). In particular C++ and Rust so the article contains useful info for that. C++ that would be more problematic I think, would have to resort to full Emscripten or something... I've tried to compile standalone C++ stdlib but wasn't much sucessful yet.
BTW, I see in the screenshots you also test Firefox, I think it would make sense to test the latest two ESR versions (these are often used by Linux distributions).
chews 2 days ago [-]
It's wild to me that PoW systems are how we sort the bot problem... the bummer is that all this "work" is just wasted cycles, at least in crypto there is a token you can sell.
ChocolateGod 21 hours ago [-]
> the bummer is that all this "work" is just wasted cycles
There's probably an overlap with people who think anubis is a good idea and those who think we should be doing more to battle climate change.
The two views are not compatible though.
Levitating 2 days ago [-]
> at least in crypto there is a token you can sell
That would defeat the purpose. The goal is to make scraping costly, not profitable.
articulatepang 2 days ago [-]
Scraping would be costly in this world: scrapers would have to spend tokens in order to get the webpage.
But, in this world, the website owner would receive tokens that they can then use to do whatever they want, including paying for servers and bandwidth. This is the sense in which the cycles aren’t wasted: the website owner now has cash to spend.
Effectively, both scrapers and ordinary users would be paying for the privilege of getting website bytes.
This also solves the problem of having to wait for your phone to solve the challenge while you’re browsing: you can buy or mine some tokens ahead of time and pay them as soon as challenged. So can the scrapers, but because they’re accessing enormous numbers of pages it’s hopefully prohibitively expensive for them.
Levitating 20 hours ago [-]
I am convinced. Is there any project implementing or proposing this?
cozzyd 2 days ago [-]
I guess it could in principle be profitable for the website, not the client?
xosc 2 days ago [-]
unfortunately at that point it would be indistinguishable from running cryptojacking on your website
LastTrain 1 days ago [-]
No. The client is doing the work.
odo1242 1 days ago [-]
The client is doing the work in both cases
TacticalCoder 2 days ago [-]
Yup was thinking the same: make honest people pay $0.00001 when they visit the site (in electricity/compute), have the challenge made so that only the website wins a tiny something. Bleed the bots dry.
> This makes Anubis challenges use a memory-hard proof of work function (argon2id) instead of just a CPU hard one. It also means that the "hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
Nice.
ancientcatz 1 days ago [-]
[dead]
saagarjha 2 days ago [-]
Sure, then just replace it with something that is useful to society but not immediately profitable to a scraper, like science research
xena 2 days ago [-]
I want to do this eventually, but it's hard to split things into the micro-tasks that would be required to make this work on Anubis. One of the ideas I'm throwing around is a world where Anubis helps fuzz old games to find timesaves in tool-assisted speedruns. It's harder than you think.
YoshiRulz 1 days ago [-]
The tools that exist for fuzzing speedruns (TASers call it "botting") are fairly primitive, excluding a few game-specific ones. Because of that, there really aren't that many TASes where a distributed randomised search could make improvements which don't get immediately overshadowed within a day of human attention. Improving botting tools and increasing their adoption would be more effective IMO. (Incidentally, TASVideos.org recently banned one of the people pioneering bot development.)
Meanwhile BOINC is well-established as the platform for distributed computation. If you can figure out how to squeeze its work units into Wasm challenges, I'm sure a lot of researchers would thank you.
thunderfork 18 hours ago [-]
BOINC isn't really built for "small tasks", afaik. Once you get small enough, the work the server is doing to manage the tasks gets to be around the same cost as the server just doing the tasks itself.
keeda 7 hours ago [-]
Yeah, I'm using PoW-based "DDOS defense mechanism" for a current project and had this thought too. I briefly looked into what it would take to exploit PoW to do something economically valuable like folding@home, and it looked a bit tricky. At first glance, these sort of projects seem a bit unsuitable for real-time PoW because they were designed to be run on desktops and so tend to do things in large batches. There probably are other use-cases that could be more suitable though.
Might be worth chucking that thought into Astra, especially if someone springs the $$$ money for it?
brendoelfrendo 2 days ago [-]
This would be rad.
account42 18 hours ago [-]
It's wild how many people happily jump on this DRM train now that they are hoping to gain from it.
odo1242 2 days ago [-]
Well, there was CoinHive which did this exact thing 6-7 years ago, but that system got abused a bit much
CarVac 2 days ago [-]
PoW was originally anti-spam technology.
anthoniks 1 days ago [-]
I think the idea is to just raise marginal costs for scraping high enough to make it expensive and non-profitable.
How much data does it download off the server to run the challenge? I've made some tiny webassembly binaries before, but I've never seen Rust generate something tiny.
xena 1 days ago [-]
It's in the range of 32-40Ki before compression. After compression it gets to be completely negligible.
stephenlf 2 days ago [-]
Comparatively, this is what it's like getting all of this working across browser versions, platforms, and so many other things:
…<img not found/>…
I don’t know what this picture was supposed to be, but a 404 demonstrates your point perfectly well.
xena 2 days ago [-]
Refresh, the Anubis docs site regenerates its cookie secret on every deploy as part of the "holy shit did I break everything?" verification step.
rfgplk 20 hours ago [-]
> It also means that the "hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
If they "vibeslopped" WebAssembly in Anubis they would have been done a year ago.
initramfs 1 days ago [-]
the enemy of my phone's temperature is my friend.
yencabulator 16 hours ago [-]
enum error {
no-error,
Huh? Isn't that covered by `result`? Sounds like you should take that out and return `result<_, error>` from `write-verification`.
shevy-java 1 days ago [-]
Somehow WebAssembly does not want to become as popular as HTML, CSS or JavaScript. Very disappointing to see.
htl 11 hours ago [-]
[flagged]
H3l0s_T3k 18 hours ago [-]
[flagged]
browserforest 1 days ago [-]
[flagged]
kevinbaiv 1 days ago [-]
[flagged]
kevinbaiv 1 days ago [-]
[flagged]
tryka7966 15 hours ago [-]
[dead]
greatgib 2 days ago [-]
[flagged]
junon 2 days ago [-]
Perhaps understanding why it exists is a good start, as opposed to flinging childish language around.
account42 18 hours ago [-]
Childish imagery begets childish language.
greatgib 2 days ago [-]
[flagged]
applfanboysbgon 2 days ago [-]
Anubis is significantly better than Cloudfare from a ux perspective because it doesn't require interacting with the page. Being gated by turnstiles and having to stop what youre doing while you're waiting is infuriating. To say nothing of Google hard blocking people it doesn't like out of websites and alluding to moving towards ID verification for its captchas.
You sound like a child throwing a tantrum. Before throwing around insults calling people post-teenager, you should maybe look at your own behavior. Is adult behavior getting angry to the point of swearing because you saw a cartoon character?
anuthis 1 days ago [-]
Argumentum ad "anime girl beats multibillion dollar company"
brendoelfrendo 2 days ago [-]
It's funny, because on the scale of things that are annoying an uncool, I find someone saying the word "retarded" as a pejorative in the year 2026 to be far, far more annoying and uncool than a harmless anime-inspired mascot for an open source project.
greatgib 1 days ago [-]
[flagged]
junon 1 days ago [-]
[flagged]
greatgib 1 days ago [-]
[flagged]
howunfortunate 1 days ago [-]
I think you'll find that 98% of normies think the opposite
bitwize 2 days ago [-]
[flagged]
1 days ago [-]
account42 18 hours ago [-]
Typical "we get to be assholes to people as long as we call them assholes first" logic.
kingstnap 2 days ago [-]
> hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
Lmao yeah no. I don't think a little argon2 is going to change shit all.
I mean the thesis of Anubis itself is "scrappers are compute limited (in ways that consumer devices are not)" which has its own massive flaws.
Jtariiiii 2 days ago [-]
The goal isn't to eliminate scrapers, its to prevent a distributed scraping network from requesting 10000 pages a second each from 10000 different websites.
lxgr 2 days ago [-]
Yes, I also strongly suspect that this is only going to move more parts of scrapers onto consumer devices. The egress proxies are already there, why not use a little bit of the compute as well?
fwip 1 days ago [-]
When you force a low end device to burn CPU or fill ram constantly, the owner throws it out and buys a new one.
mattstir 21 hours ago [-]
That's just not possible in many parts of the world.
bakugo 1 days ago [-]
Haha, no. The average tech illiterate person tolerates a LOT when it comes to bad performance on low end devices.
fwip 16 hours ago [-]
The malware already results in noticeable slowdown just running the scraper. Solving anubis challenges is significantly more difficult than simply requesting the page, so it is not feasible on-device without making the device totally unusable.
Rendered at 10:30:03 GMT+0000 (Coordinated Universal Time) with Vercel.
Love this. There’s been past discussion on HN re how OSS maintainers are treated, and this is such a wry sentence. I really appreciate the tone / attitude to the problem.
I'd say "wide variance in the level of kindness" is a rather British way to describe some of the opinions that have been voiced.
So this is going to be interesting.
My plea (in a small voice) to the OP: just as many sites still do for JavaScript, please add a "This captcha required Webassembly to continue" message to your code when Webassembly is disabled. And, incidentally, for smaller platforms/browsers that don't offer a Webassembly engine at all.
[1] https://webassembly.org/docs/use-cases/
> One of the big things that blocked this shipping for so long was not having an escape hatch of some kind to allow clients that disable WebAssembly by policy to get through the gate. [...] This combination of factors means that there would need to be another implementation of the proof of work code in JavaScript that would actually execute the number crunching.
Point of order: WASM doesn't let things run in the background. To do that you need WebWorkers (a javascript feature, not WASM-specific).
I mean, you get the same indicators as when the page runs arbitrary Javascript on load? WASM assembly may be (slightly) harder to read than typescript compiled to JS, but the source is all there, it calls the same APIs, etc.
I forgot to do so on my phone. I noticed Firefox had slowed to a friggeb crawl when opening pages. Disabling web workers fixed that.
My wish is actually for the days of NPAPI where Webassembly would be just another plugin (like Flash) and browsers clung to mainly HTML, JS, and CSS. Then developers wouldn't be quite so keen to rely on it and the web would be much less bloated. Yes, those days are long gone... As a tradeoff, I'd settle for a simple toolbar icon that lit up when Webassembly is running, and/or a high level UI (just like Location, Camera, and Microphone) so I can enable it for just those sites that need it. I tried selling Mozilla on the idea once but they apparently weren't interested.
Baselines aren't entirely universal. I remember the days when the Mozilla (or early Firefox) browser was the only viable web client for Linux, but some sites still opted to run behind ActiveX. Today I don't think you'll find a Haiku or Amiga browser with Webassembly support, nor Firefox for Linux on IBM Power motherboards. It's easy to forget that just because most any 'nix or BSD OS has a plethora of choices, that others aren't still dealing with the "ActiveX" situation and feeling like everybody's out to get them.
You do see how selfish it is, right? While the author did introduce such a solution, it's such a massive waste of time.
This amounts to giving Google effectively complete control over the web. I'm not sure that's a good idea. IMO it's important that Firefox and Safari remain popular enough that webdevs are forced to support them.
I'm somewhat sympathetic to "If you want to 'customize' your web experience, you shouldn't be surprised when things don't work right." in terms of turning off commonly-used features, but the request isn't to make it work right. OP just wants a helpful error message.
I have a Mac from 2014 running Yosemite that I occasionally use to test for backwards compatibility in my own frontend code (for fun!). But IMO the best way to ensure compatibility is to use period-correct toolchains or toolchains where the pace of change is slower, like ClojureScript.
Fun fact, this _used_ to be the case - wasm32-unknown-unknown had extra non-mvp features added later, which in my eyes has been a breaking change on stable. You're not the first person to have been bitten by this, there's [1] and a similar story in Ruffle [2].
Weirdly, the response from the rustc side (IIRC we also talked with one of the devs of wasm component on discord a bit around 2024) has contained a lot of (paraphrasing) "in retrospect we think wasm32-unknown-unknown target was designed badly*, so we'd prefer if you either dealt with it or switched to another wasm target <with its different set of downsides> than us fixing it". (*I acknowledge that the way std was implemented on that target was unusual and "not clean", but it works _really well_ for a lot of people.)
> I looked into the process involved for rebuilding the standard library twice: once with only MVP wasm features enabled and once with an "all yes config" like usual. Based on some research I did this seemed like a massive pain.
That's what Ruffle does [3], it's a bit unfortunate but wouldn't call it a massive pain.
[1] https://internals.rust-lang.org/t/can-we-have-either-a-new-t...
[2] https://github.com/ruffle-rs/ruffle/pull/18397
[3] See lines 55 and 117-127 in https://github.com/ruffle-rs/ruffle/blob/6890c618daab3153343...
That's about right. Specifically, wasm32-unknown-unknown is serving multiple targets, which aren't very compatible, and it implements std by panicking in all the bits that can't work.
One target is "you truly have absolutely nothing, and don't really have much of `std` at all". Another target is "you're in a browser, and can do anything a typical browser can provide" (ideally targeting the future where wasm has full browser APIs without going through javascript). Another target is "hosted environment that isn't a browser, and which capabilities you have depends on the environment". Those are three very different things.
I would argue that we need at least two new targets, possibly three:
- `wasm32-none` or similar (by analogy with `x86_64-unknown-none`), a target where you don't have anything except for what you bring with you. The equivalent of writing kernel code.
- `wasm32-wasi-hosted` (or pick another name), where you the hoster of the wasm can supply whatever capabilities you have available.
- `wasm32-wasi-browser`.
It's possible that `none` could be a special case of `hosted`, where the services you provide are "none".
On https://wasm-feature-detect.surma.technology it shows that I don't have 3 of all these features but I'm not sure if Anubis needs any of them to not kick me back to the pure JS solution
Which would apparently be bad because
> The WebAssembly that's shipped with this flow is ridiculously performant. This may mean you need to adjust the difficulty [upward, to avoid that bots solve it trivially when they support wasm, I assume this means]
combined with
> The wasm2js flow doesn't currently have a way to update the progress bar [so you have no clue about remaining time]
(wouldn't be the first time that I gave up on a page because it was stuck on 0 hashes per second)
And while I’m sympathetic to the idea of not wanting to run JS, to a first approximation modern browsers are JS engines that have graphical displays. How things should be vs how they are is a classic is/ought problem. The world took a vote on what a browser’s meant for and we lost. Fighting it today is rough; tomorrow, futile.
No, because there are technologies that don't have this issue, eg. privacy pass.
Whenever I hit Anubis, I simply go "keep your secrets then" and take my leave. The vast majority of the sites posted to HN (that catch my fancy) work fine or fine enough, and are better for it.
If I really, really desperately want to check something out, I can always just turn stuff back on. Turns out, I rarely do.
Shoutout to the particularly dogshit few that don't just require cookies and JS, but even third party JS. For reading a blogpost or a message thread!
They probably meant 3rd party cookies. It's common to block those. 1st party cookies not so much.
I feel like Anubis is ironically speedrunning a lot of discoveries the crypto folks have already made several years ago...
We could link it to a site, you generate for HN, I for Reddit, but it so happens that you visited Reddit more and I HN, so we depleted our Anubis POW, so we could exchange some Reddit Anubis with some HN Anubis.
I don't understand the units here, wouldn't a bit versus nibble difference make the multiplier 16, instead of 1024?
Also, while some pages may look like static HTML pages, they may be generated on the fly by an expensive and/or slow backend, which adds to the cost. I happen to maintain servers for academics and some content management systems are slow and have an expensive CPU cost. While it's OK for the low number of humans interested in the subjects they deal with, it's definitely not fit for massive bot scrapping. And before you ask, no, it's not always practical to have cache upfront or to pre-generate all pages.
When I see LKML using Anubis when the pages it serves are tens of kB, all presumably static, I wonder if they do it out of spite rather than to protect their servers.
As a user/reader/viewer I absolutely hate Anubis and usually turn around when I see it pop up (at least on my phone where it takes ages to compute), but with stats like that, I get why a site operator would resort to using it.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
14 CPU cores total for running a website like kernel.org is laughable. This is not worth burning cycles in Anubis on client's devices, this is not worth the time of the engineer who worked on it. Provisioning more hardware would have been literally better for everyone.
This drives me crazy with so-called SOTA LLMs that have "achieved AGI".
Fable, Sol, Astra, will start by trying to reverse engineer a binary to figure out how something works when software is open source and one search query away.
You let them know it's open source, and they will start using github API instead of just cloning and grepping.
The situation you are in is far less dire sounding when you consider that you have these options available to you.
Well, sounds like it's not your problem then.
Yes and that should be fixed before you subject real users to resource-wasting scripts.
And to answer your intended demand: this is in some cases impossible or unreasonable. And things were working fine before LLM DDoS.
It also depends on if you want your content to be scraped. Any bot ignoring robots.txt should be considered potentially malicious and worth considering such mechanisms for.
That's a trivially solved problem - have the web server refuse (or redirect if you're feeling nice) requests with unexpected query parameters. No reason to make things worse for real users before optimizing your site.
And even if there are actual parametric resources you want to serve, apply limits only to them and let users visit the rest without unnecessary bot checks.
I could probably also analyze all the paths across my servers and write some bespoke Nginx config files to filter out all requests with illegal queries, but that's harder, more fragile, and just leaves the servers open for the next time a scraper finds a new way to screw me over.
The bots I'm fighting have gone for the nuclear option, relying on botnets and spoofing to bypass basic filtering, so I don't see why I shouldn't do the same.
Also, if you are not using a big CDN (e.g. Cloudflare), you still need to pay for the traffic (even if you don't pay as much for the CPU). This is doubly true for smaller CDNs like Bunny.
I only hear bad things about bots lately, are there any benevolent bots?
I've done a tiny bit of scraping myself (usually to archive old websites before they die), and I've gotten banned a few times even though I wasn't hammering anything.
Looks like it's getting harder and harder to do scraping for legitimate purposes. Although I suppose legitimate is subjective :) Everything becomes valuable when it's old enough...
The git host example is probably the one which Anubis is the dumbest defense for: the main reason to have those links is for easy machine interaction. So that's dead when you implement it.
So why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.
As you said, if you actually want to interact with a remote git repo you'd do so with packfiles not diffs.
I document my code, in issue and elsewhere on the Web, by referring to specific commits so having a URL for each is useful to me at least.
Would there be any risk in making a dynamically built algorithm (i.e. the risk that we don't find N zero prefix bits)? Chain a set of transforms and build a unique WASM binary (including optimizations to deter pattern matching); possibly one a day or hour or something. That way you'd need to have Claude vibeslop the CUDA implementation at that cadence, per website.
Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves
Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing
Scrapers can't change their strategy? It's not hard to think of mitigations: keep on using a session until you're banned, and failing that rotate your IP. Nowadays you don't even need to think. You can probably tell a LLM "write me a scraper that has adaptive anti-anti-bot mechanism".
Yes, but you normally don't try to fight all strategies up front, you add stuff on a need-to basis. Currently, lots of people are in the stage of "need to get this thing using bunch of IPs to stop hammering my servers", apparently Anubis helps them with that. If that stops working, other tactics are tried, and so it goes on for the life of your service. Hasn't stopped yet, doesn't seem to be stopped soon either.
Turns out that in practice there's some bots that implement support for this but not JavaScript.
People have been sent to jail for participating in LOIC attacks which were less destructive that what these bots do on the daily.
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
This takes many minutes to solve on a cellphone, 9 milliseconds on a cheap VPS. It’s not stopping anyone.
You could just spend 30 seconds googling instead of confidently asserting nonsense. There are plenty of people demonstrating that Anubis reduces bot traffic.
"Anubis reduces bot traffic" and "[the proof of work] is not stopping anyone" isn't contradictory, especially if anubis isn't being benchmarked against non-POW alternatives like IP/ASN bans, or tls fingerprinting.
Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages.
There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.
If against all odds it turns out that skiddies (or agents or AI billionaires or what have you) are willing to burn inordinate amounts of RAM and CPU just to scrape autogenerated cgit pages that are of approximately zero value to them to begin with then I guess we'll just have to cross that bridge when we come to it.
And then regular users: who now can't really browse your site.
And then smart bots - anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down.
The solution basically makes the favored customer well funded AI scrapers, not users.
Extreme configurations with high difficulty do make for a frustrating experience on very old devices, but in practice it rarely gets in the way.
Only the tail end of what one could consider normal users (those running custom browsers or having JS disabled or nonfunctional) run into issues here, but they'll have these issues with any other bot-blocking mechanism that still offers some protection these days.
Apple's "ask your vendor for tokens to exchange for access" web standard gets around this, assuming those bespoke and Javascript-less browsers implement PATs, but that standard seems to be completely dead in the water outside of Safari.
That's entirely up to how the site operator configures the PoW challenge.
> anyone with even a moderate amount of funding in the AI space, who aren't even slightly slowed down
Do you have any evidence that it is the large well funded AI companies DoSing sites by (for example) scraping absurd numbers of permutations of cgit url parameters?
Even if it is (which I doubt), if all this accomplishes is getting them to spend 10 minutes optimizing their bots to not do that it will be a major win.
- https://techaro.lol/robots.txt
- https://anubis.techaro.lol/robots.txt
and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.
I assume that the previous challenges will still be available for circumstances where WASM is not available (unless we are just working on the basis that it is available if JS is?) or perhaps as a fallback. TFA explicitly mentions “smart” TVs, and I'm sure there are plenty of those out there that run browsers old enough to not know what WASM is. TBH these would not be a priority for me, but they obviously are for this project given they (well, TVs generally, not specifically elderly ones) get specific mention in this write-up.
If so, what is to stop scrapers from just forcing the fallback POW option instead of trying to run the new memory-hard algorithm at all?
I've just finished creating a WASM 1.0 implementation (simple JIT on x86 and x86_64) for plugin usage and I'm looking on how to support more languages beyond C (for which I have "micro-libc" to create really small binaries). In particular C++ and Rust so the article contains useful info for that. C++ that would be more problematic I think, would have to resort to full Emscripten or something... I've tried to compile standalone C++ stdlib but wasn't much sucessful yet.
BTW, I see in the screenshots you also test Firefox, I think it would make sense to test the latest two ESR versions (these are often used by Linux distributions).
There's probably an overlap with people who think anubis is a good idea and those who think we should be doing more to battle climate change.
The two views are not compatible though.
That would defeat the purpose. The goal is to make scraping costly, not profitable.
But, in this world, the website owner would receive tokens that they can then use to do whatever they want, including paying for servers and bandwidth. This is the sense in which the cycles aren’t wasted: the website owner now has cash to spend.
Effectively, both scrapers and ordinary users would be paying for the privilege of getting website bytes.
This also solves the problem of having to wait for your phone to solve the challenge while you’re browsing: you can buy or mine some tokens ahead of time and pay them as soon as challenged. So can the scrapers, but because they’re accessing enormous numbers of pages it’s hopefully prohibitively expensive for them.
> This makes Anubis challenges use a memory-hard proof of work function (argon2id) instead of just a CPU hard one. It also means that the "hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
Nice.
Meanwhile BOINC is well-established as the platform for distributed computation. If you can figure out how to squeeze its work units into Wasm challenges, I'm sure a lot of researchers would thank you.
Might be worth chucking that thought into Astra, especially if someone springs the $$$ money for it?
https://github.com/eternal-flame-AD/pow-buster - Browser extension that already used WASM to accelerate the anubis solver, among others.
https://github.com/kasper93/anubis_webgpu - Browser extension that uses webgpu, for another order-of-magnitude speedup (depending on your GPU of course).
…<img not found/>…
I don’t know what this picture was supposed to be, but a 404 demonstrates your point perfectly well.
If they "vibeslopped" WebAssembly in Anubis they would have been done a year ago.
You sound like a child throwing a tantrum. Before throwing around insults calling people post-teenager, you should maybe look at your own behavior. Is adult behavior getting angry to the point of swearing because you saw a cartoon character?
Lmao yeah no. I don't think a little argon2 is going to change shit all.
I mean the thesis of Anubis itself is "scrappers are compute limited (in ways that consumer devices are not)" which has its own massive flaws.