Speculated to be responsible for the "AI restaurant-menu gore" that made the rounds on social media. Restaurants AI-generate the entire menu sheet from whole cloth, and while it looks "good" at first with modern image models, repeated edits to the menu text/layout transform the passable food "photos" into trypophobia-inducing horrors. (And somehow they don't notice or care)
Notably, the recent ChatGPT Images 2.5 announcement claimed to address this issue specifically ("multi-turn editing consistency")
Sharlin 14 hours ago [-]
With local models it's always been easy to edit only a part of an image (inpainting), either by painting a mask or automatically using a small segmentation model. It's actually pretty strange that commercial models have resorted to redrawing the entire image even in cases where the requested change is localized.
GPerson 1 days ago [-]
Does the same effect happen in a society which rewrites all of its knowledge over and over using AI?
vidarh 21 hours ago [-]
Depends. With or without quality control? Not any more than in a society that rewrites all of its knowledge over and over using using humans. We even have a term for doing so without quality control: A game of telephone.
Though you might well argue that the risk of us dropping the quality control gets higher the cheaper it gets to churn things out with AI.
tencentshill 9 hours ago [-]
It's why any "synthetic training data" startup is doomed to fail.
rcxdude 13 hours ago [-]
I would argue it's a risk of any process that loops around itself without grounding in reality periodically.
camillomiller 21 hours ago [-]
Definitely already happening with many c-suites.
We unleashed Nyarlatothep…
“I do not recall distinctly when it began, but it was months ago. The general tension was horrible. To a season of political and social upheaval was added a strange and brooding apprehension of hideous physical danger; a danger widespread and all-embracing, such a danger as may be imagined only in the most terrible phantasms of the night. I recall that the people went about with pale and worried faces, and whispered warnings and prophecies which no one dared consciously repeat or acknowledge to himself that he had heard. A sense of monstrous guilt was upon the land, and out of the abysses between the stars swept chill currents that made men shiver in dark and lonely places. There was a daemoniac alteration in the sequence of the seasons—the autumn heat lingered fearsomely, and everyone felt that the world and perhaps the universe had passed from the control of known gods or forces to that of gods or forces which were unknown.
And it was then that Nyarlathotep came out of Egypt”
abecedarius 1 days ago [-]
This reminded me of a story/vignette which my fallible memory tags as by someone like Twain or Poe, about an automatic rewriting machine. It was put to work on the same passage with an ever higher intensity setting, becoming more and more gothic to the point of parody -- iirc the final output was verse including the line "to atoms dash the doubly damned".
Does anyone remember this?
nkurz 1 days ago [-]
No memory, but searching says it was Lewis Carroll in Photography Extraordinary, published in 1885:
The Milk-and-Water School
Alas! she would not hear my prayer!
Yet it were rash to tear my hair;
Disfigured, I should be less fair.
She was unwise, I may say blind;
Once she was lovingly inclined;
Some circumstance has changed her mind.
The Strong-Minded or Matter-of-Fact School
Well! so my offer was no go!
She might do worse, I told her so;
She was a fool to answer "No".
However, things are as they stood;
Nor would I have her if I could,
For there are plenty more as good.
The Spasmodic or German School
Firebrands and Daggers! hope hath fled!
To atoms dash the doubly dead!
My brain is fire--my heart is lead!
Her soul is flint, and what am I?
Scorch'd by her fierce, relentless eye,
Nothingness is my destiny!
It’s not that bad though. I expected way worse. If Robin Hobbs wrote it, I’d barely lift a brow.
Muromec 1 days ago [-]
It doesn't merely stabilise, it almost converges on itself
MarkusQ 1 days ago [-]
It would have been interesting to try something closer to the movie poster example; say have the AI rewrite the passage with the room toggling between the initial light-mode decor and a dark-mode decor, then back, repeatedly. Or even change the woman's name to "Mrs. Smythe" and back.
layer8 1 days ago [-]
As the article explains: “[…] In other words, if you instruct a model to change a single word in a paragraph of text, it can almost always handle the task with no collateral damage.“
MarkusQ 13 hours ago [-]
Right, but if you ask it to make a systematic change (the room decor for light to dark and back) or something with subtle implications that would require other changes to fit in?
Changing the title in the movie example is this sort of change; the "change a single word" isn't analogous.
groby_b 1 days ago [-]
The existence of attractors (e.g. "stable at iteration 31") for AI models is one of the weirdest things in AI land. They are a wonderful space to explore, fascinating and full of horrors.
Meanwhile, my hat's off to the (currently) last comment on the page: "I like how the shear-sheathed legs still earned their keep — probably load-bearing for Mrs. Regan"
vunderba 14 hours ago [-]
Related 2025 paper on how GenAI running in an unconstrained loop tends toward convergent collapse:
Thanks, that's more like it! Even includes an Art Deco mirror...
jrochkind1 1 days ago [-]
> PS. For obvious reasons, this post will fail Pangram tests.
Actual pangram to me says it's 100% human. Current pangram can flag different passages differently, but these are all flagged human by it, even the LLM written ones.
brzezmac 17 hours ago [-]
that's why I argue the LLMs are just a bunch of glorified lossy compression engines ;)
rob74 15 hours ago [-]
In the case of the movie poster, it probably doesn't help that the image was apparently stored in a lossy format. So after a few iterations the LLM can't tell the JPEG artifacts from the relevant image details and starts adding wrinkles to the faces and texture to the clothes etc. until it all goes to hell...
xiaoyu2006 17 hours ago [-]
Be cautious of that claim. Lossy compression may exactly hold intelligence.
sophrosyne42 12 hours ago [-]
Probably not, since our intellect can produce lossless compression too
roger_maddux_iv 1 days ago [-]
[dead]
rkeswick 18 hours ago [-]
[dead]
michaelmrose 1 days ago [-]
[dead]
amelius 1 days ago [-]
The same happens with humans, in e.g. chain stories, cumulative stories, etc.
Not unique to AI.
lowbloodsugar 1 days ago [-]
>I don’t write about generative AI much. Everyone else does; not much of consequence ever gets said.
I see he chose not to break with tradition.
camgunz 21 hours ago [-]
Sing it with me now, "these things aren't inteeeeeeeelligeeeeeeeent"
Rendered at 04:05:04 GMT+0000 (Coordinated Universal Time) with Vercel.
Notably, the recent ChatGPT Images 2.5 announcement claimed to address this issue specifically ("multi-turn editing consistency")
Though you might well argue that the risk of us dropping the quality control gets higher the cheaper it gets to churn things out with AI.
“I do not recall distinctly when it began, but it was months ago. The general tension was horrible. To a season of political and social upheaval was added a strange and brooding apprehension of hideous physical danger; a danger widespread and all-embracing, such a danger as may be imagined only in the most terrible phantasms of the night. I recall that the people went about with pale and worried faces, and whispered warnings and prophecies which no one dared consciously repeat or acknowledge to himself that he had heard. A sense of monstrous guilt was upon the land, and out of the abysses between the stars swept chill currents that made men shiver in dark and lonely places. There was a daemoniac alteration in the sequence of the seasons—the autumn heat lingered fearsomely, and everyone felt that the world and perhaps the universe had passed from the control of known gods or forces to that of gods or forces which were unknown. And it was then that Nyarlathotep came out of Egypt”
Does anyone remember this?
https://archive.org/details/further-nonsense-verse-and-prose...
Changing the title in the movie example is this sort of change; the "change a single word" isn't analogous.
Meanwhile, my hat's off to the (currently) last comment on the page: "I like how the shear-sheathed legs still earned their keep — probably load-bearing for Mrs. Regan"
https://pubmed.ncbi.nlm.nih.gov/41583980
initially https://share.gemini.google/eJ54qDXM1gzC
boldly and maximally improved tone, clarity and flow: https://share.gemini.google/RiqX4x6mLB82
boldly improved: https://share.gemini.google/cR72HXmY4UdH
Actual pangram to me says it's 100% human. Current pangram can flag different passages differently, but these are all flagged human by it, even the LLM written ones.
Not unique to AI.
I see he chose not to break with tradition.