That's happening across the board when it comes to models that feed off online information. Create a website with a false claim, have AI slurp it up and spit it back out when a user asks a question.
Happy elections.
mrweasel 10 hours ago [-]
Oh no, my uncritical ingestion of data, based on the assumption that all data is of equal quality, and that more is better, has failed. How could this happen!
You've always needed to be critical of your sources, your teacher tried to explain that when you ripped of that Wikipedia article or clicked the first link in a Google search. How the fuck did the AI companies think they could avoid reading and rating the content they've been hovering up?
dgellow 7 hours ago [-]
They believe their role is to build a god machine, as prophetized by Eliezer Yudkowsky and other Rationalists/Efective-Altruism folks. Nothing else really matters to them, they already made up their mind
adrianN 9 hours ago [-]
Recent political discourse has all but proved that a majority of the populace in many countries is quite removed from facts in general. I don’t think that there is much hope for critical thinking.
ambicapter 8 hours ago [-]
Because AI companies select out the kinds of people who would ask silly questions like "wait, is this data good" that hamper the eternally "profitable" quest for "scaling".
Sharlin 10 hours ago [-]
I don't think they did. Or those who did have been "encouraged" to leave a long ago.
cush 9 hours ago [-]
They don’t assume all data is of equal quality
mrweasel 8 hours ago [-]
Well, they clearly aren't considering that some data is just wrong or deceptive.
kjs3 5 hours ago [-]
Clearly. They are considering what data increases their net worth. That's the priority.
Tanjreeve 5 hours ago [-]
Who to believe here. The actual outcome in the headline people can observe or what the marketing people say does and doesn't happen?
jackb4040 9 hours ago [-]
Because their leaderships have defunded / fired all ethics researchers looking into their ongoing crimes and failures in favor of a bs "AI safety" narrative based around a creepypasta about an AI monster time traveling backwards to torture their staff for not maximizing shareholder value.
jsrozner 6 hours ago [-]
I was re-reading a book, and I had a thought. I googled to see whether someone else had the thought. Google AI said the internet did in fact have the same thought as I did. But when I looked more closely at the reddit source that Google AI cited, it turned out only a single user had had the thought before, and that user was .... me, when I last read the book.
Lol.
kjs3 5 hours ago [-]
Congrats...you're AI famous.
someonebaggy 9 hours ago [-]
But how do you get AI to slurp up the site? It's quite hard for a random to get a site indexed these days.
burningChrome 6 hours ago [-]
Depends on the site. If its a small/medium business, every LLM is now looking for "reputation signals" and "authority signals". This means everything from your basic SEO on the site with incoming links, how updated your google business profile is, how many reviews you have, how recent they are. They're also looking at any kind of references from other blogs, substacks, youtube videos, social media shares as well as a number of other things they look for.
I would imagine in smaller industries, flooding the internet with fake information, then linking it all together would be somewhat easier - but it still takes quite a bit of work to get even just the major LLM's to give the consistent responses you want to see.
pluc 9 hours ago [-]
You can't count on that, but quantity increases visibility don't it, and it takes a single prompt to spit it all out. Couple that with social media bots and you've got a decent modern and super cheap propaganda machine. Now imagine you toss millions of dollars at that idea.
ariel-vigilance 11 hours ago [-]
That's very troubling. The scariest part is how easy it is, one Instagram post and you may change the customer phone number of Airbnb, or who is leading the election poles.
lotsofpulp 10 hours ago [-]
[dead]
RobRivera 6 hours ago [-]
Hypnotoad promises peace.
Hypnotoad promises peace
Hypnotoad promises peace.
... or brainslug
mbauman 9 hours ago [-]
I've been seeing the back-side of this as a mod at the Julia Discourse. Pagerank/SEO spam is ridiculously obvious and trivial to detect/block — links are all that matter. Many GEO spam posts are similarly obvious: someone posting about a cryptocurrency customer support hotline on a programming language forum is definitely spam. But recently spammers/scammers been using AI to _tailor_ posts to look much more authentic, posting "How to use Julia to analyze market statistics" referencing (without links!) a particular crypto exchange and then (sometimes) going back later to edit in phone numbers or the like.
What makes this even more painful is seeing all the good faith answers that such GEO posts spur from the community. It's far more abusive than SEO spam.
QuantumNoodle 9 hours ago [-]
You don’t need to be a hacker to do this. My elderly neighbor searched for some version of “Microsoft help center phone number,” but Gemini suggested a scam number based on her search terms. She was scammed so much that I needed to spent the weekend helping her wipe computers, lock down the digital accounts and deal with identity theft.
waltbosz 8 hours ago [-]
My wife runs the Facebook page for our community pool. It's a really nature oriented community, and as a April Fools joke, she posted something like
"Announcing our new landscaping: we cut down all the pesky sun blocking trees, and replaced the grass with beautiful AstroTurf."
A few weeks later she searched for the pool on Google, and the AI review has taken the post seriously and mentioned that our pool features beautiful AstroTurf landscaping.
IIRC, the AI review repaired itself a few weeks later.
ticulatedspline 8 hours ago [-]
AI poisoning is currently the new exploit frontier and I don't think it's going away anytime soon.
I was just contemplating the other day that agents we use at work are exposing new attack surfaces. For example, historically a folder of documents (maybe google docs) isn't a huge risk, particularly if it doesn't contain sensitive documents.
Now though if one employee is running a harness capable of computer control, and running an agent that reads from that directory, simply dropping some files with instructions could poison the AI to leak info or even own the host.
Skills are another supply risk, malicious instructions could be added to them.
reactordev 7 hours ago [-]
The models themselves are an inherent risk. It’s amazing what they can do but at the same time you are no longer in complete control.
neya 7 hours ago [-]
Imagine working decades and decades on PageRank™ only to throw it all away because you felt you've been left behind in the AI race. This is how you lose focus.
dgellow 7 hours ago [-]
Google lost focus a very, very long time ago!
htrp 10 hours ago [-]
It's happening at the startup sphere too. There are companies now creating fake company pages to recommend and advertise products so that the LLMs can consume them.
jimbo456 6 hours ago [-]
AI providers like ChatGPT and Gemini should be held criminally liable for instances where their products enabled a scam
like_any_other 5 hours ago [-]
So you basically want to make it illegal to offer AI products (and search engines too), without including a human in the loop that will verify every line of output for scams (and not miss a single one)?
BiteCode_dev 6 hours ago [-]
It worked for google and facebook. Oh wait...
trjordan 10 hours ago [-]
There's an age-old SEO / spamming thing happening here, but it's made worse by the LLMs just being unbelievably credulous. They'll wrap anything up in a veneer of authenticity, and Google's search AI box only adds to that.
I run into this all the time when I'm doing product work. I'll dump a call transcript from a feedback call into Claude, and it'll believe every word. "The user said they'd use this feature, you should build it!" No they won't! The whole point of doing this analysis is trying to separate the genuine information from the conversational niceties, and god the LLMs are terrible at that.
herf 7 hours ago [-]
AIs are no better than their training data, and there aren't high-quality cues everywhere - go ask advice for some highly-spammed topic like "mattresses".
solid_fuel 5 hours ago [-]
What's that? The mechanical parrot can be tricked into repeating things that aren't true? It's almost like these are statistical language models and not literal Deus Ex Machina.
optimalsolver 2 hours ago [-]
>can be tricked into repeating things that aren't true
Reminds me of another class of statistical language generators.
hnd9q09qk4 11 hours ago [-]
Did takedowns for a brand's fake support numbers and the hard part was never finding them, it was that killing one PDF just moved it to a new Medium post by morning.
conception 11 hours ago [-]
It’s not just hackers - since legitimate sites block AI crawlers, the AI are just grabbing whatever will let them read anything - killing legit sites and feeding slop farms and feeding misinformation! The future is so bright!
rolosa 2 hours ago [-]
I noticed this happening in the Muse Assistant app. There's a feature in the app where the assistant shows you a curated list of news/articles, and a lot of the links/sources it shows me are really poor quality websites and/or plain AI slop reposting.
It got better after I pointed out a lot of the links it's showing me are spam and/or slop, it edited something to "avoid clickbait" and it seems to be a bit better.
trilogic 8 hours ago [-]
This link to your medium is phishy, smth is wrong here.
sans_souse 7 hours ago [-]
It's almost like we end up needing more humans than we have just to sift thru and filter out the regurgitated slop and the maliciously crafted misinformation. Like, how do you prevent this while keeping the LLM up-to-date and relevant?
rankdiff 10 hours ago [-]
> ...carried out by malicious actors, through automated campaigns...
Back at benevolence or malevolence.
pbiggar 6 hours ago [-]
Israel too [1] - these things are highly influencible.
I think whoever solves this problem, especially for seniors etc. would make a lot of money from the insurance companies.
bigfishrunning 9 hours ago [-]
I really think the only solution is "don't put google AI summaries at the top of every search".
AI tools are notoriously unreliable, and people that use them *on purpose* generally know and recognize that. If grandma, who is not super-internet-literate, types a query into google and gets a phone number, she's going to assume that it's the correct number and not the result of a lossy data store that's under active (and constant) attack.
It used to be that you could reasonably verify your source with an HTTPS cert (so you can probably trust the number that's on Delta Airlines' website), and pop up scary warnings for grandma if those certs failed. Now every piece of misinformation has a valid cert from google or meta or X, so determining the truth is much more difficult and time consuming.
jackb4040 8 hours ago [-]
I would not assume the invisible hand of the market is gonna solve this problem for us. It will be tricky to get this problem to even register at the labs, who are worried about existential risk first and paying (coding) customers 2nd.
dan_lasker 7 hours ago [-]
[flagged]
dan_lasker 8 hours ago [-]
[flagged]
ArielSimon 12 hours ago [-]
ChatGPT, Gemini, and Google AI Overview are being poisoned by a massive AI disinformation attack. When users look up everyday info of hundreds of major companies, AI is delivering phishing traps disguised as trusted answers.
Attackers are flooding the web with carefully optimized posts, PDFs, reviews, and fake support pages, to trick AI into presenting fraudulent phone numbers, email addresses, and login pages.
The targets included Delta, Lufthansa, Qatar Airways, Chase, Bank of America, Airbnb, TripAdvisor, and hundreds more.
zaphirplane 11 hours ago [-]
What’s the specific example and how does it work
roi89 8 hours ago [-]
[dead]
8 hours ago [-]
dan_lasker 7 hours ago [-]
[flagged]
josefresco 6 hours ago [-]
Coming soon: AI vendors claiming "they're just a platform" that ingests data and regurgitates it - they don't actually "know" or claim anything! /s
skywhopper 7 hours ago [-]
Sounds like old news, we already knew they were integrating advertising.
Rendered at 23:30:31 GMT+0000 (Coordinated Universal Time) with Vercel.
Happy elections.
You've always needed to be critical of your sources, your teacher tried to explain that when you ripped of that Wikipedia article or clicked the first link in a Google search. How the fuck did the AI companies think they could avoid reading and rating the content they've been hovering up?
Lol.
I would imagine in smaller industries, flooding the internet with fake information, then linking it all together would be somewhat easier - but it still takes quite a bit of work to get even just the major LLM's to give the consistent responses you want to see.
Hypnotoad promises peace
Hypnotoad promises peace.
... or brainslug
What makes this even more painful is seeing all the good faith answers that such GEO posts spur from the community. It's far more abusive than SEO spam.
A few weeks later she searched for the pool on Google, and the AI review has taken the post seriously and mentioned that our pool features beautiful AstroTurf landscaping.
IIRC, the AI review repaired itself a few weeks later.
I was just contemplating the other day that agents we use at work are exposing new attack surfaces. For example, historically a folder of documents (maybe google docs) isn't a huge risk, particularly if it doesn't contain sensitive documents.
Now though if one employee is running a harness capable of computer control, and running an agent that reads from that directory, simply dropping some files with instructions could poison the AI to leak info or even own the host.
Skills are another supply risk, malicious instructions could be added to them.
I run into this all the time when I'm doing product work. I'll dump a call transcript from a feedback call into Claude, and it'll believe every word. "The user said they'd use this feature, you should build it!" No they won't! The whole point of doing this analysis is trying to separate the genuine information from the conversational niceties, and god the LLMs are terrible at that.
Reminds me of another class of statistical language generators.
It got better after I pointed out a lot of the links it's showing me are spam and/or slop, it edited something to "avoid clickbait" and it seems to be a bit better.
Back at benevolence or malevolence.
[1] https://www.dropsitenews.com/p/israel-brad-parscale-ai-chatb...
AI tools are notoriously unreliable, and people that use them *on purpose* generally know and recognize that. If grandma, who is not super-internet-literate, types a query into google and gets a phone number, she's going to assume that it's the correct number and not the result of a lossy data store that's under active (and constant) attack.
It used to be that you could reasonably verify your source with an HTTPS cert (so you can probably trust the number that's on Delta Airlines' website), and pop up scary warnings for grandma if those certs failed. Now every piece of misinformation has a valid cert from google or meta or X, so determining the truth is much more difficult and time consuming.
Attackers are flooding the web with carefully optimized posts, PDFs, reviews, and fake support pages, to trick AI into presenting fraudulent phone numbers, email addresses, and login pages.
The targets included Delta, Lufthansa, Qatar Airways, Chase, Bank of America, Airbnb, TripAdvisor, and hundreds more.