Once I was at a company which both had agreements to rehost certain content, and scrape the significant amount.
A couple of funny things happened, once we were clearly being blocked and my manager wanted me to figure out a way to evade said blocking.
And in another case we ended up getting scraped ourselves. Which was absolutely hilarious. I think we were able to resolve this by just contacting the website host.
I am not a lawyer, but a lot of things in America are almost 100% dependent on how much money you want to spend suing or defending yourself in court.
There are a lot of VC funded companies who more or less just scrape and resell data. If LinkedIn and few other places figure out how to actually block them , they'd be out if a business.
armchairhacker 23 hours ago [-]
The recent DOSs are not scraping. You can scrape a website running on a hobbyist home server without inconveniencing it, even 1000 people daily can. The AI "scrapers" sending multiple requests per second (even per day) to every imaginable URL (vs. sitemap.xml or robots.txt) should be more accurately called DOSers, and are intentionally malicious and/or ridiculously incompetent.
djoldman 1 days ago [-]
This one made me chuckle:
> This breaks my students’ brains. If it’s so ubiquitous, it must be clearly legal…right?
Do they not drive? Driving slightly faster than the speed limit is ubiquitous in my experience.
slyall 24 hours ago [-]
I'll guess you are from living in the USA
In other countries going over the speed limit a small amount will get you fined and your license suspended if you do it by enough or often.
People in countries like Australia have to warn US visitors that they have to keep below the speed limit
account42 9 hours ago [-]
Maybe you should not generalize your own knowledge to "in other countries". I no country where I have lived do you get fined for a measured speed slightly above the posted speed limit. The difference may be mainly there to account for measuring imprecision but effectively it means you can get away with a bit over the speed limit.
slyall 1 hours ago [-]
Well I guess where you live going 15km/h over the speed limit is accepted like in many parts of the USA.
However in general most 1st world countries have more strict enforcement of speeding than the US does. So the speeding laws are not a good example of unenforced.
ranger_danger 32 minutes ago [-]
> However in general most 1st world countries have more strict enforcement of speeding than the US does
Do you have a source to back up this claim?
aksss 23 hours ago [-]
> "enough or often"
Yes, there is an "enough or often" limit in the USA as well. State driver's licenses are based on a point system and too many infractions will result in a suspension of your license if your points dip below a given threshold. Your margin of speed vs limit increases the point deduction. Fines vary down to individual localities, some of them being annoyingly high. In most places (I believe) driving with a suspended license is a criminal offense.
One big difference may be that most places in the USA have been able to resist photo radar/LPR automation of issuing speeding tickets, so the odds generally favor you in most areas until they don't. There's also this tacit allowance for "going with the speed of surrounding traffic". If all the traffic on the freeway is going 10MPH over the limit, you're generally safe in the herd like a zebra.
The USA is a big place with a very diverse set of driving conditions and jurisdictions, so the more you know an area the better you can navigate the character of that area's enforcement. Some places you'll deal with flagrant enforcement pushes at the end of the month as police try to meet quotas. Some places use traffic law enforcement to subsidize their budgets. Our mapping apps can alert us to speed traps if other drivers report them. Drive an area long enough, you'll know where the common speed traps are, and understand the patterns in road design that facilitate them. Generally, we have the luxury of being able to flout speeding laws to such a degree that we can feel indignant when we're caught, but the law is the law.
brendoelfrendo 1 days ago [-]
Well, as he goes on to say, a scraping campaign must (based on current law, anyway) satisfy a fairly specific list of constraints to be considered unambiguously legal. That's where the "not clearly legal" idea comes in; I can only imagine that law students would have trouble parsing the idea that a massive tech company with billions of dollars would fail to spend even a little bit of that to evaluate whether or not their scraping was legal and just... do it anyway.
ls612 1 days ago [-]
This is a feature not a bug in the system. But for a lot of idealistic young law students learning that will make them deeply uncomfortable
moffkalast 1 days ago [-]
Ask a cop pulling you over if it's a feature as he writes you a ticket lmao.
eli 1 days ago [-]
It certainly is to the cop. How else could they do a pretext stop?
ls612 1 days ago [-]
That was precisely my point. “Show me the man and I’ll show you the crime” is the fact of how western legal systems work, the rule of law is a tale that they tell to impressionable schoolchildren.
aksss 23 hours ago [-]
It's certainly not unique to western legal systems, and I'd even say not generally(!) a characteristic of them. That quote comes from the Soviet Union and is more associated with totalitarian regimes where nebulous thought crimes are on the books and laws spun out of whole cloth. Think China.
Sure, when the state wants you, they'll get you. Read Three Felonies a Day: How the Feds Target the Innocent. I'm not one to say otherwise (quite the opposite), but for most people, most of the time, in most of their lives, western legal systems are pretty top notch in protecting people from arbitrary and corrupt application of the law.
In systems that truly embrace arbitrary and corrupt enforcement, it's quite a bit different system than largely objective traffic laws which society has agreed upon but which are rarely enforced (e.g. turn signals, lane merging in intersections, etc.) which turn into selective enforcement. I say this as someone who firmly believes that selective enforcement is ripe for abuse and has inherent problems, but on the balance I prefer the latitude I have as a driver in today's system than one where all traffic regs are enforced through automation at scale (which may be where we're headed in the next few decades, honestly). The reality is that as drivers, we probably curse the fact that these laws aren't enforced more often until we're the ones cited.
ls612 21 hours ago [-]
And I’m saying that three felonies a day and the way China does it are fundamentally the same thing with different window dressing.
aksss 14 hours ago [-]
Perhaps only in a very abstract sense of the power of any state dwarfing that of an individual, but not in any practical or “as-applied” sense. The reality of the general individual experience in China vs the US legal system is a very stark contrast (which we can get into but is also an easy AI convo).
After all, when you refer definitionally to “the legal systems of the west”, to what alternative cohesive system of legal theory are you contrasting it to? There aren’t many, and they have differences we can measure. For example, those emphasizing divine revelation (Sharia) or subordination to the state interest (Chinese legal thought). Leave personal value judgements aside - it would be incorrect to think these systems operate the same way in practice or are “fundamentally the same”.
Outside of the few cohesive systems of theory and tradition that we can identify, the alternatives are systems characterized by capriciousness and corruption - the absence of the rule of law.
ls612 8 hours ago [-]
I’d suggest that one of the key innovations of authoritarian control over the populace in the 21st century vs the 20th is that surveillance and compute means the authoritarian states don’t need to use the brute force of the legal system against the average person to keep power. They can focus only on key targets while not molesting the broader population in the same way the Stasi et al did. I’d go so far as to say for the typical American and the typical Chinese their experience of the legal system is virtually identical, in that they don’t experience it. But for those targeted by the system, the precise mechanism may vary but the certainty of the outcome does not.
aksss 4 hours ago [-]
I can see your point of view, though "those targeted by the system" implies a lot of circumstance and conditions.
It's very true that both citizens' experience with their legal system changes drastically when state interests and national security are implicated. Like China, western legal systems recognize national security and public order as compelling state interests.
The difference is that an individual in China is far more likely to find themselves at an intersection of state and political interest with their individual activity (be it art, speech, action, association, religion, property) than one would find in the west. In China, the NSL structurally defines political security and cultural and social order as being within the national security framework. That's intentional and sits in contrast with the western tradition where the state generally needs to justify restrictions on protected individual conduct with a claim of state interest, and those claims are frequently subjected to an adversarial review process.
As you say, technology has already made surveillance far more intrusive and effective than anything available in the 20th century. Of course, the effect of the Stasi wasn't simply the surveillance, it was the uncertainty of never knowing if your colleagues, neighbors, or even family could be personally trusted - that environment of uncertainty was an intentional social control element. In a similar but not identical way, uncertainty is leveraged in the Chinese legal system: one may not necessarily know when their activity (or even past activities) have suddenly fallen on the wrong side of the national security line. This variability is not a bug in the system but a deliberate feature, and stands in contrast with the western legal systems.
Once entangled, there are bright line differences in the structure of those systems - opportunities to appeal are far fewer in China as a foundational difference, but opportunities to challenge the basis of the laws themselves and the justification for a claimed state interest essentially border on being non-existent. In contrast, individuals or classes of individuals bringing facial or as-applied challenges to overturn regulation or a claim of compelling state-interest is a frequent occurrence in US (for instance).
If we consider it a given that the state naturally seeks greater control of individuals (and I do), then the western legal system and particularly the US legal system, has been uniquely effective at constraining the state. In China, for example, it's not even a consideration that CCP leadership can be curtained through administrative proceedings.
But there's no arguing that in the US, if the stakes are high enough, they'll "get their man". On the flip side, the surveillance of the state itself by the citizens has also restricted the ability of the government to act with as much impunity as they did, say, sixty years ago. Specifically, policy body cams, ubiquitous cell phone video, one-party consent laws, legal clarity on recording public officials in public spaces, FOIA laws, etc. Nonetheless, it's disturbing and frustrating how inscrutable some very public events remain.
ls612 3 hours ago [-]
And this is different in what substantive way to the western system? The vast vast majority of ordinary Chinese will never interact with their national security state. This reads as you saying I’m right without saying I’m right.
wand3r 1 days ago [-]
Honestly interesting article and general topic. As with most things it depends on who you are. If you are Spotify, Google, OpenAI or Anthropic, its fine. As mentioned, this is why gray areas and unsettled laws are so malicious. They protect the rich and powerful and can be used to absolutely destroy regular people. It is adjacent to scraping, but Aaron Schwartz is a good example. He faced decades in jail for what is effectively best practice for an LLM data pipeline
tzs 22 hours ago [-]
> It is adjacent to scraping, but Aaron Schwartz is a good example. He faced decades in jail for what is effectively best practice for an LLM data pipeline.
None of what Swartz was charged with [1] is anywhere near best practices for an LLM data pipeline.
(Also, he wasn't facing decades in jail. That's from the DoJ press release which uses a completely stupid way of calculating potential sentences [2] that gives very high numbers so it always make it sound like they have caught some super villain).
I don't think that there are laws against LLM distillation in the United States. It's only a terms of service violation. For all of the complaints about model distillation that OpenAI and Anthropic make, you don't see them saying that it's against the law:
They would probably like it to be against the law ("Distillation attacks at this scale require a coordinated response across the AI industry, cloud providers, and policymakers") but I hope that legislators decline to grant their wish.
eli 1 days ago [-]
I don't see why that should necessarily be true.
bfung 24 hours ago [-]
If you read the article, scraping is “not clearly legal”. Read those 3 words carefully.
ang_cire 23 hours ago [-]
Distillation is legal. ToS violations are not crimes. There's nothing illegal about capturing output to train with it.
There will probably be (hypocritical) AI companies trying to claim some kind of minimal "protections" are applied to the output in order to misuse DMCA violation claims, but it's BS.
AIWithLucas 11 hours ago [-]
[flagged]
builder_woody 12 hours ago [-]
[flagged]
becketthayes 11 hours ago [-]
[flagged]
FinnAnderson 11 hours ago [-]
[flagged]
EtienneDeLyon 21 hours ago [-]
[dead]
Rendered at 22:51:28 GMT+0000 (Coordinated Universal Time) with Vercel.
A couple of funny things happened, once we were clearly being blocked and my manager wanted me to figure out a way to evade said blocking.
And in another case we ended up getting scraped ourselves. Which was absolutely hilarious. I think we were able to resolve this by just contacting the website host.
I am not a lawyer, but a lot of things in America are almost 100% dependent on how much money you want to spend suing or defending yourself in court.
There are a lot of VC funded companies who more or less just scrape and resell data. If LinkedIn and few other places figure out how to actually block them , they'd be out if a business.
> This breaks my students’ brains. If it’s so ubiquitous, it must be clearly legal…right?
Do they not drive? Driving slightly faster than the speed limit is ubiquitous in my experience.
In other countries going over the speed limit a small amount will get you fined and your license suspended if you do it by enough or often.
People in countries like Australia have to warn US visitors that they have to keep below the speed limit
However in general most 1st world countries have more strict enforcement of speeding than the US does. So the speeding laws are not a good example of unenforced.
Do you have a source to back up this claim?
Yes, there is an "enough or often" limit in the USA as well. State driver's licenses are based on a point system and too many infractions will result in a suspension of your license if your points dip below a given threshold. Your margin of speed vs limit increases the point deduction. Fines vary down to individual localities, some of them being annoyingly high. In most places (I believe) driving with a suspended license is a criminal offense.
One big difference may be that most places in the USA have been able to resist photo radar/LPR automation of issuing speeding tickets, so the odds generally favor you in most areas until they don't. There's also this tacit allowance for "going with the speed of surrounding traffic". If all the traffic on the freeway is going 10MPH over the limit, you're generally safe in the herd like a zebra.
The USA is a big place with a very diverse set of driving conditions and jurisdictions, so the more you know an area the better you can navigate the character of that area's enforcement. Some places you'll deal with flagrant enforcement pushes at the end of the month as police try to meet quotas. Some places use traffic law enforcement to subsidize their budgets. Our mapping apps can alert us to speed traps if other drivers report them. Drive an area long enough, you'll know where the common speed traps are, and understand the patterns in road design that facilitate them. Generally, we have the luxury of being able to flout speeding laws to such a degree that we can feel indignant when we're caught, but the law is the law.
Sure, when the state wants you, they'll get you. Read Three Felonies a Day: How the Feds Target the Innocent. I'm not one to say otherwise (quite the opposite), but for most people, most of the time, in most of their lives, western legal systems are pretty top notch in protecting people from arbitrary and corrupt application of the law.
In systems that truly embrace arbitrary and corrupt enforcement, it's quite a bit different system than largely objective traffic laws which society has agreed upon but which are rarely enforced (e.g. turn signals, lane merging in intersections, etc.) which turn into selective enforcement. I say this as someone who firmly believes that selective enforcement is ripe for abuse and has inherent problems, but on the balance I prefer the latitude I have as a driver in today's system than one where all traffic regs are enforced through automation at scale (which may be where we're headed in the next few decades, honestly). The reality is that as drivers, we probably curse the fact that these laws aren't enforced more often until we're the ones cited.
After all, when you refer definitionally to “the legal systems of the west”, to what alternative cohesive system of legal theory are you contrasting it to? There aren’t many, and they have differences we can measure. For example, those emphasizing divine revelation (Sharia) or subordination to the state interest (Chinese legal thought). Leave personal value judgements aside - it would be incorrect to think these systems operate the same way in practice or are “fundamentally the same”.
Outside of the few cohesive systems of theory and tradition that we can identify, the alternatives are systems characterized by capriciousness and corruption - the absence of the rule of law.
It's very true that both citizens' experience with their legal system changes drastically when state interests and national security are implicated. Like China, western legal systems recognize national security and public order as compelling state interests.
The difference is that an individual in China is far more likely to find themselves at an intersection of state and political interest with their individual activity (be it art, speech, action, association, religion, property) than one would find in the west. In China, the NSL structurally defines political security and cultural and social order as being within the national security framework. That's intentional and sits in contrast with the western tradition where the state generally needs to justify restrictions on protected individual conduct with a claim of state interest, and those claims are frequently subjected to an adversarial review process.
As you say, technology has already made surveillance far more intrusive and effective than anything available in the 20th century. Of course, the effect of the Stasi wasn't simply the surveillance, it was the uncertainty of never knowing if your colleagues, neighbors, or even family could be personally trusted - that environment of uncertainty was an intentional social control element. In a similar but not identical way, uncertainty is leveraged in the Chinese legal system: one may not necessarily know when their activity (or even past activities) have suddenly fallen on the wrong side of the national security line. This variability is not a bug in the system but a deliberate feature, and stands in contrast with the western legal systems.
Once entangled, there are bright line differences in the structure of those systems - opportunities to appeal are far fewer in China as a foundational difference, but opportunities to challenge the basis of the laws themselves and the justification for a claimed state interest essentially border on being non-existent. In contrast, individuals or classes of individuals bringing facial or as-applied challenges to overturn regulation or a claim of compelling state-interest is a frequent occurrence in US (for instance).
If we consider it a given that the state naturally seeks greater control of individuals (and I do), then the western legal system and particularly the US legal system, has been uniquely effective at constraining the state. In China, for example, it's not even a consideration that CCP leadership can be curtained through administrative proceedings.
But there's no arguing that in the US, if the stakes are high enough, they'll "get their man". On the flip side, the surveillance of the state itself by the citizens has also restricted the ability of the government to act with as much impunity as they did, say, sixty years ago. Specifically, policy body cams, ubiquitous cell phone video, one-party consent laws, legal clarity on recording public officials in public spaces, FOIA laws, etc. Nonetheless, it's disturbing and frustrating how inscrutable some very public events remain.
None of what Swartz was charged with [1] is anywhere near best practices for an LLM data pipeline.
(Also, he wasn't facing decades in jail. That's from the DoJ press release which uses a completely stupid way of calculating potential sentences [2] that gives very high numbers so it always make it sound like they have caught some super villain).
[1] https://volokh.com/2013/01/14/aaron-swartz-charges/
[2] https://web.archive.org/web/20130211030902/https://www.popeh...
https://openai.com/index/disrupting-a-coordinated-model-dist...
https://www.anthropic.com/news/detecting-and-preventing-dist...
They would probably like it to be against the law ("Distillation attacks at this scale require a coordinated response across the AI industry, cloud providers, and policymakers") but I hope that legislators decline to grant their wish.
There will probably be (hypocritical) AI companies trying to claim some kind of minimal "protections" are applied to the output in order to misuse DMCA violation claims, but it's BS.