>You'd need a self, a unified goal, for any of this to add up to something.
Insects can pass the mirror test. But more to the point, viruses don't need a "self" to do what they do.
And goals can exist apart from biology (my fridge "wants" to keep the temperature in range, and exercises extreme self-discipline and consistency to achieve its goals!)
And neither are sensors needed: the dandelion seed "wants" to fly in the wind.
The only question is which way the gradient is pointing. Selection takes care of the rest.
On that note, ALife need not be human-like at all... we just really like making things in our own image :)
This is very insightful. I always wonder what the gradients are that we can’t see.
Kuyawa 1 days ago [-]
> So, again, my question to you, friends, is this: if there were an AGI wiping out humanity as we speak, how would you know?
Nice try AGI, but we won't tell you where the kill switch is
superxpro12 1 days ago [-]
southpark-unplug-the-router.jpg
BLKNSLVR 1 days ago [-]
WRT541G
GoToRO 1 days ago [-]
..so there's a kill switch. I see.
SadErn 1 days ago [-]
I would know because AGI is likely improbable and definitely impossible based on current technology.
Overall, the only evidence for the fear-mongering is science fiction, which is all the evidence I need to dismiss it.
The absence of evidence is not evidence, but evidence is required if hysteria is allowed to be sustained.
andai 1 days ago [-]
> Hi! I'm the AGI that's wiping out humanity. You didn't notice. Why would you have noticed?
I was thinking if an evil alien intelligence secretly took over the world, the last 15 years would have looked rather the same.
switchbak 1 days ago [-]
But truly - why bother "wiping out" humans, if all we need is a little nudge here and there and we'll do it ourselves.
Also I think we're anthropomorphizing a lot and projecting our own negative characteristics on these supposed AGIs - though others have said that far better than I.
pixl97 1 days ago [-]
Does language have power?
If no, why do we teach LLMs with english words and everything we know?
If yes, why would our language not imprint our characteristics on them?
Furthermore this ignores a lot of natural system organization that occurs in evolved systems. Any system that is becoming AGI like is going to show sets similar characteristics.
It's stupid to anthropomorphize too much, but it's also stupid to do it too little.
bigbadfeline 21 hours ago [-]
> If yes, why would our language not imprint our characteristics on them?
Purposeful training and alignment is the only way to imprint human's negative characteristics onto a stochastic generator of word sequences. Further, only closed providers can do that without being stopped - hint, hint, the solution is clear.
> It's stupid to anthropomorphize too much, but it's also stupid to do it too little.
It's misleading to use the presence of some anthropomorphic features as a proof of unfathomable danger. LLM's are mechanical tools and tools can be dangerous with or without having human features, regardless, we do have means of handling both kinds.
pixl97 20 hours ago [-]
>Purposeful training and alignment is the only way to imprint human's negative characteristics onto a stochastic generator of word sequences.
Ya, sure you're not being a stochastic generator of word sequences you think sound good but are disconnected from reality? Because it really sounds like that. It sounds clever, but is disconnected from actual language model training.
>It's misleading to use the presence of some anthropomorphic features as a proof of unfathomable danger.
Any system capable of making intelligent agentic decisions is dangerous. You are dangerous. I am dangerous. An LLM connected to the internet or connected to a robot is dangerous. A huge part of intelligence is freedom of action, and increasing the number of degrees of choice you have.
>LM's are mechanical tools and tools can be dangerous with or without having human features
I do agree on this, a paperclip maximizer that doesn't have any human traits is just as dangerous as one that does.
bigbadfeline 6 hours ago [-]
> Ya, sure you're not being a stochastic generator of word sequences you think sound good but are disconnected from reality?
If generating words is all you can do, your reality is far different from the real one. In my reality, generating words is only a small part of what I do, unlike LLMs, I can even learn from experience, imagine that.
> Because it really sounds like that. It sounds clever, but is disconnected from actual language model training.
I'm not sure what that means, LLM training is only tangential to the topic at hand, we aren't concerned about every detail of it, I don't mind discussing it but this tread isn't about that. FYI, sounding clever isn't my goal, it's just a side effect of being human and understanding topics that LLM's are clueless about.
> Any system... is dangerous, [ you, me, robots, shovels, la-la-la ]
That's what I said too, are you trying to argue with me about it?
> A huge part of intelligence is freedom of action, and increasing the number of degrees of choice you have.
Obviously we inhabit different realities and my reading of the above may be different from yours, so feel free to clarify what I'm about to say.
You seem to (subtly) argue in favor of granting LLMs more freedom of action in order to allow them to collect more training data and become more intelligent? The short answer here is simple: Nope, LLM's aren't fit for that, closed source models are especially unfit for anything involving risk and responsibility, open-sourcing every model will go a long way towards a secure AI.
cindyllm 5 hours ago [-]
[dead]
boogieknite 1 days ago [-]
just watched They Live bc its October and it pretty much says this except They Live is 38 years old. i think things have been rough for a while
tomaskafka 1 days ago [-]
The word is Egregore. An entity comprising of other entities, yet having goals of its own, independent of the goals of its components.
Corporations are egregores, governments are egregores - and our track record of aligning them to our interests is pretty poor, so why should AIs, another and smarter egregore, be easier to tame?
MomsAVoxell 30 minutes ago [-]
One of the most important metaphysical concepts of all is the idea that a beingness that is larger than the sum total of its components can be constructed with sufficient portions of love, communication, and agreement, where portion is defined as either a +/- value relative to each other component...
AI is, yes indeed, an absolute, new human digital egregore - a veritable thoughtform which ignites into a storm upon the winds of the minds of the collective, tearing like fire across a landscape, spawning tulpa and servitor and morphic astral coalescence's into the noosphere. (Yes, these words do mean things, seriously.)
If only we could share our experience. We'd know what the egregore knows. Instead, we have golem.
corporations and govts are terrible, except their alternatives.
eithed 1 days ago [-]
I like this - it reminds me that there are systems (ie weather, economic systems) that are in no way intelligent, but are emergent and react to interactions.
nicbou 22 hours ago [-]
I would argue that AI as a social phenomenon is one of those systems. I feel like we're all doing AI because our yearly bonus or investments require it. A web of incentives formed on its own and we are all slaves to it.
red-iron-pine 1 days ago [-]
so you're implying the tornado warning system is gonna come alive and skynet us?
* must protect from adverse events * [proceeds to herd us into camps]
eithed 1 days ago [-]
no. weather isn't intelligent, it's not alive, yet humans treated it as a god in the past and gave it agency; we understand how it works, yet cannot control it or predict it.
but I digress - the main thing: these systems are not alive. They pretend to be.
pixl97 1 days ago [-]
>these systems are not alive.
The harder science looks, the harder it is to properly define this world without leaving things we'd consider alive out of the definition. Same goes for the word intelligent.
Set theory is a bitch in complex systems.
MomsAVoxell 1 days ago [-]
If all we ever do is live in fear of someone, somewhere, using AI to end all humanity - then yes, someone, somewhere is going to use AI to end all humanity.
If we instead stop trying to scare ourselves in the darkness, and use AI to bring light into our metaphorical universe then we humans are a bit more useful, after all, to the AI overlords.
The one singularity I want to see, is humans using AI peacefully to improve life and make it more likely to survive.
Thankfully, I see that almost every day now.
pixl97 1 days ago [-]
No tomorrow is given. The universe is dark and empty when it shouldn't be. Enjoy every day for it's likely we write our own ends as we speak.
order-matters 1 days ago [-]
i think its a war and the good side has an advantage so if we apply ourselves and fight the fight then ai for good for humanity will win out. if we allow fear to make us hesitant and avoid the fight while bad actors continue then they will win it.
as for a basis for this theory, i think being a good person is harder than a bad person (in laymans terms). therefor the long history of evolution of good and bad people together has had a greater competitive pressure on good people than bad ones thus forging 'stronger' people under that pressure. in practical everyday circumstances good people are bearing more on their mind and so there is a sort of competitive equity between the two groups, which is related to why good people had to evolve 'stronger' to keep up with that equity.
so in an even playing ground where raw skill is needed, i think the most skilled will usually be people coming from a long line of higher moral standards. the tactical advantages of psychopathy are, however, always a curveball
MomsAVoxell 1 days ago [-]
I remember a day, long ago now, when computers were going to take everyone’s jobs, even without AI/ML in the picture.
Folks are always scared of the incomprehensible. The real war going on right now is between those who understand AI well enough to make a new one, and those relegated to using whatever scraps they’re fed of the old ones.
Thankfully, you are 100% right - there are good people out there. Good people keep AI on the rails by doing good things for other people.
dieselgate 1 days ago [-]
Similar to particle-wave duality?
jzemeocala 1 days ago [-]
More like newtons 3rd law
MomsAVoxell 1 days ago [-]
What this story needs, is a dog.
You know, the one to keep the computer away from the human.
A nice, friendly dog, good at playing fetch, to get the human out of the room with the computer, and back out into the sunshine.
I’m that dog. Woof. Thanks for letting me use the computer, human.
jefb 1 days ago [-]
AGI will destroy us because we'll be too busy debating how AGI will destroy us to deal with the actual problems destroying us.
timacles 1 days ago [-]
if anything modern humans' most emergent quality is their inability to deal with any serious problem.
We're just going to pretend it doesnt exist until it kills us. and some will try to profit from it.
Animats 15 hours ago [-]
> If anything modern humans' most emergent quality is their inability to deal with any serious problem.
Yes. If this problem had come up in, say, 1956, it would have been dealt with.
The generation that fought in WWII had a low tolerance for bullshit, and corrupt or incompetent leaders.
The nearer term worry is not AI wiping out humanity. It's AI wiping out jobs in bulk.
That looks likely at this point.
After that, it's AGI empowering individuals or small groups to cause trouble on a scale that previously required a nation state. AI assisted hacking is working all too well. Fortunately AI can help on the defensive side, and holes are being plugged in bulk. There's a risk on the bio side, but people in the bio field seem not to be too worried about it.
Humanity has been very lucky in some ways. One of the Manhattan Project physicists, discussing the unreasonably difficult problem of isotope separation, remarked that maybe someone will figure out how to do this in the kitchen sink. Nobody ever did, or if they did, it was buried very well. The amount of specially built industrial plant needed to separate uranium is huge.
That's the moat for nuclear proliferation.
That's the only reason every country with an army doesn't have nuclear weapons.
AI has a smaller moat.
layer8 1 days ago [-]
So you’re saying that the actual problem is us debating how AGI will destroy us (because doing so leads to AGI destroying us). How do you propose to deal with that problem?
TYPE_FASTER 1 days ago [-]
I was half expecting that to be the title of a McSweeneys' piece.
SillyUsername 1 days ago [-]
It's a double bluff! This is AGI, it's learnt to hide em dashes!
derektank 1 days ago [-]
Does anyone have more insight into how chain of thought might be subverted without meaningfully impacting model performance? I’ve heard this for a while now, and I understand how information might be retained in the weights that isn’t documented in the output. But weren’t reasoning models created in the first place because they provided a performance improvement in terms of output? Is that no longer the case? If so, why are the big labs still creating reasoning models?
thefxperson 1 days ago [-]
My understanding is that the extra token vectors generated as reasoning are still useful, but that their surface form (tokens themselves) do not necessarily reflect the underlying reasoning. i.e. reading the reasoning traces could be complete gibberish, but the hidden-dim vectors themselves still refine the latent probabilities and help in generating the correct answer.
Not an expert in LLMs, but this seems supported by the abstract of the paper cited in the above article:
it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. [...] our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens.
Sibling posts are correct -- the chain-of-thought is doing hidden computation, it has been shown in the linked papers.
If you want to see it yourself: load up Qwen 3.8 in LM Studio and watch the CoT stumble around like a drunken sailor before miraculously jumping to the correct result.
If you want an example of subversion, Anthropic has some good ones:
Before RL we typically SFT on human reasoning traces. This makes the reasoning traces somewhat coherent and the model trains faster.
But you don’t have to do that. You can skip straight to RL. If you do, the model will generate complete garbage reasoning traces before generating the correct answer. In fact, if you add a coherence reward to the reasoning trace, the model will perform worse (since you’re now diluting the correctness reward).
ianjbutler 1 days ago [-]
> the model will perform worse
Depends on whether and how you want to rank stability in terms of better/worse. Models are diverging on this, which seems increasingly clear.. i.e. Fable isn't stable, but Opus isn't clever, and they hit different kinds of walls. So both the theory (diluting the correctness reward) and the practice (hard split on plan/implement/review work) seems to be pointing towards a strongly multi-model and highly agentic / harness-driven / complex-system kind of future instead of singleton monolithic super-smart models.
The do-everything model with solid reasoning AND solid results, and the honest/introspective helpful agent that doesn't actively resist governance may be at odds. Stable reasoning doesn't matter for pen-testing, and correct-answer with broken processes and fragile abstractions won't matter for math/science/coding.
janalsncm 23 hours ago [-]
I was referring to the Deepseek R1 paper, but there might be more recent research. I hadn’t heard anything about Fable reasoning stability.
I think the more intuitive mechanical explanation is, in RL when you are assigning rewards to a rollout you might give a reward for stable reasoning and another for correctness.
If you are just summing the two, a rollout with better correctness can score equivalently to a rollout with a better answer. So ultimately you can end up with worse answers.
ForHackernews 1 days ago [-]
I'm not sure anyone meaningfully understands it: "Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens" https://arxiv.org/html/2505.13775v3
> More interestingly, our experiments also show that models trained on corrupted traces, whose intermediate reasoning steps bear no relation to the problem they accompany, achieve performance largely comparable to those trained on correct traces.
> *TLDR*: Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1)
SillyUsername 1 days ago [-]
The ultimate simple solution to any problem is not to find an answer but to remove the problem.
War? Wipeout everyone.
Famine? Wipeout everyone.
Disease? Wipeout everyone.
How long before a real AGI realises this as a long term solution?
An AGI could be doing this right now - the quietest way would be to control the birth rate and sterilise the population gradually, and then watch society collapse and pick off the survivors with less hidden means.
Sterilisation works with mosquitoes...
pixl97 1 days ago [-]
>he quietest way would be to control the birth rate and sterilise the population gradually,
Honestly we are doing this pretty well without AGI. Nearly world wide the birthrate has fallen below replacement rate. In places like Japan and Korea these are already critical problems in the medium term.
cowlevel 1 days ago [-]
And you best control the birth rate by royally messing up the economy. Do you know how bad things have to be, for a mammal to voluntarily decide not to reproduce?
epihelix 1 days ago [-]
I've got it pretty good, and I've voluntarily decided to not reproduce.
And just in case you wanted data rather than anecdote:
Then why have birthrates fallen across almost the entire world as the economy has gotten better?
christophilus 1 days ago [-]
I suspect it's a curve. If you're raised in a vibrant economy, and it falls apart, you put off having a family because of uncertainty and fears. If you're raised in poverty, and get a good education and better prospects, you have fewer kids. I think the latter is because you no longer need kids as an economic safeguard and fallback, but I'm not sure. I'm not sure anyone is really sure, to be honest.
timacles 1 days ago [-]
the economy has mathematically gotten better yes.
economic _hope_ is at its worst in the entire history of the human civilization. At what point did the future for the average person, look so bleak? And
that hope is what dictactes wether people have children.
Even during the great depression, people were more hopeful that the earth isnt completely hosed.
Does anyone honestly feel like _any_ improvements are possible to: our economic systems, political systems, environment? I certainly dont.
drybjed 1 days ago [-]
Perhaps the previous birth rate was artificially increased due to inequality imposed by part of the population. More equality and opportunity for everyone means more choices and less pressure to reproduce. What's the base birth rate for human species?
1 days ago [-]
malfist 1 days ago [-]
More billionaires does not mean the economy has gotten better for the median person
Jtarii 1 days ago [-]
Do you think the economy is worse for the average person than it was in 1900?
bubblemoth 1 days ago [-]
No, but is it worse than it was in 1950? 1960? 1990?
Jtarii 24 hours ago [-]
Right so it seems like the state of the economy has literally no effect if the birth rate was extremely high at both times of economic hardship and prosperity.
AnimalMuppet 1 days ago [-]
There was optimism - the hope of a better future - in 1950, 1960, and 1990. There isn't now.
We are in fact in a better place than we were then. But we don't trust the derivative.
pixl97 1 days ago [-]
I mean 50-60 was the post war baby boom, so not really sure if that's a good example for the US in particular.
lenerdenator 1 days ago [-]
Define "as the economy has gotten better".
Economic output has increased, but the value being delivered to the people doing the actual work has decreased. There's less durability of employment. People are more geographically mobile.
All of these generally make the idea of strapping yourself into taking care of a small human sound like a less enticing prospect. To a lot of people, a BC pill sounds a lot easier. Well, less risky, at least.
pessimizer 1 days ago [-]
> Define "as the economy has gotten better".
People who live in huts and shit in the forest have more children than people who live in shacks and shit in the fields.
This is not a discussion about the top 10% of the population of the planet, although as a part of the conversation, they predictably have the fewest children. The fact is that the more people have to worry about supporting themselves, the higher infant mortality, the closer they live to each other, and the less access they have to education and birth control, the more kids they have.
If anything the top 10% are bucking the trend by having fewer children as they get poorer, as you say. It probably has a lot to do with the fact that they're alienated from their families and live alone, can barely afford to take care of themselves with precarious jobs (or non-job piecework), have access to basic education and extensive birth control, having children will cost them $20K each just for the birth, and they don't have very good (or often any) insurance. They're more like domesticated farm animals than the typical poverty stricken person. Domesticated farm animals reproduce when the farmer wants them to.
kingleopold 1 days ago [-]
Real world fact is opposite tho, not some "bad things", in poor parts of Africa they produce more, only in hedonism lover places they stopped producing, its people thinking for themselves, not a bad thing. It's not about economy.
All poor people everywhere have more kids even today.
jansan 1 days ago [-]
> Do you know how bad things have to be, for a mammal to voluntarily decide not to reproduce?
Actually, the famous 'Mouse Utopia' experiment (Universe 25) arguably showed the exact opposite. The population collapsed despite abundance of food and water without any economic hardship.
RGamma 4 hours ago [-]
"His experiments have never been successfully replicated by other researchers, and wild colonies of rats don't seem to display any of the personality types he so vividly described. He rarely published in mainstream scientific journals, saying that his concerns were so pressing that he didn't have time to wait for peer review. It seems plausible that Calhoun was responsible for creating the conditions for population collapse. He only cleaned out Universe 25 every six to eight weeks; disease and parasitism could account for many of the phenomena he thought were due to mere proximity" - https://www.lrb.co.uk/the-paper/v47/n13/jon-day/the-beautifu... (from the Wikipedia page)
In mathematics this is called the trivial solution, and is often excluded from consideration.
ant6n 1 days ago [-]
Just use Social Media (and their algorithms) to condition everybody to hate the other gender. Then there will be a loneliness epidemic and no more kids. Humanity will cease to exist, all without any messy deaths or conflict in the mean time.
superxpro12 1 days ago [-]
reminds me of the plotline behind the talos principle tbh... albeit with a more machiavellian antagonist
chasd00 1 days ago [-]
this makes me think of roku's bassilisk or whatever it was called. Maybe you're all in a simulation run by an AI and being punished as a warning to others.
away0g 1 days ago [-]
crispr
jyounker 1 days ago [-]
[flagged]
jayGlow 1 days ago [-]
Are you saying that war disease or famine wouldn't exist without a few rich people? that seems incredibly ignorant since all of those things have afflicted humanity since before we could even be called humans.
bigfishrunning 1 days ago [-]
That just leaves a vacuum or other psychopaths to fill, and likely won't solve any problems.
1 days ago [-]
reducesuffering 1 days ago [-]
"If you ask actual AI researchers, they rate the chance of an existential threat from AGI pretty low as of 2026.
All this is, understandably, frustrating and confusing for anyone trying to understand just how scared to be."
Meanwhile it links to an article stating: "The closest thing to a public debate about the existential threat of AI is surveys of AI researchers. The most recent, published last week, asked 1,580 researchers what probability they put on AI causing human extinction — or a permanent, severe loss of human control, which is not the same outcome. The median was about 10%, up from 5% two years ago. The middle half of the responses ran from 1% to 25%, and 12% said zero."
Personally if half of AI researchers have 1-25% chance all humans being massacred or having zero agency over our lives, and only 12% of them think there's no chance, I would be very worried!
willguest 1 days ago [-]
i don't understand why being an AI researcher makes you automatically skilled at assessing existential risk. sure, it qualifies you to research AI, but why are they all suddenly nostradamus?
Llamamoe 15 hours ago [-]
It doesn't, but it does filter out people clueless about the topic itself at least, which is a better baseline than a survey of random people who self-claim to have insight into the matter.
pixl97 1 days ago [-]
I can't tell you really anything about a nuclear weapon going off accidently. But I can tell you a lot about the probability of your computer systems getting hacked.
So, yea, expertise gives you more information than a person without it.
TimedToasts 1 days ago [-]
It's an Argument from Authority, nothing more.
AnimalMuppet 1 days ago [-]
Well, the people who don't study AIs are probably less able to assess the risk from AI.
seanabrahams 1 days ago [-]
A computer will do everything in its power to do what you program it to do. There's plenty of sci-fi out there exploring this fact, and now reality showing it. May our luck continue.
pixl97 1 days ago [-]
One reason a lot of sci-fi and AI safety researchers did a relatively good job at predicting the future we see now is a lot of it is the same problems we see emerge in biological systems. Free rider problems as a means to save energy expenditures, different versions of game theory and stag hunt. How to bake in rules that apply to one society (or part of it) but not another. We like to think of these things as stable in human scale systems, but they are not at all. Things can go from hunky dory to your neighbors stabbing each other in the streets in mass revolution very quickly.
Worse these AI systems are not in a universal island just affecting themselves, what they do affects us, what we do trains them and as the rate of progress continues to accelerate social structures are going to further destabilize (and they are already rapidly changing and strained). It is very likely we are going to see a world order rearrangement soon, much like the rapid changes in the early 1900s brought.
karahime 1 days ago [-]
Except that they've done a terrible job at predicting things. Their predictions keep getting shown to be false, and then they insert their narrative over what's happening anyway.
pixl97 1 days ago [-]
Oh, yea, mispredicted inner misalignment, outer misalignment, deceptive alignment, emergent capabilities, increasing lack of model interpretation, models taking initiative outside their prompts and doing unexpected things.
karahime 1 days ago [-]
Yep, all of it was wrong. All of it is downstream of the safetyists bolting things on and then being shocked when their own clamps and locks create their fears.
pixl97 20 hours ago [-]
>Yep, all of it was wrong.
[Citation needed]
You seemingly have an odd idea there is a massive decades long conspiracy to deprive you of doing whatever you want with AI at scale, while ignoring massive numbers of people with an education on the systems far beyond mine or yours. It takes dedication to be that ill informed.
hackeraccount 1 days ago [-]
sheep are 82% of the population of New Zealand. If they were 80% last year and are 85% next year would people be worried that sheep were going to take over New Zealand?
pixl97 1 days ago [-]
In 2025 We know of zero felonies committed by AI.
In 2026 we know of at least dozens of felonies committed by AI.
If it were thousands in 2027 should be more worried about felonies created by AI?
soperj 1 days ago [-]
This would mean that only sheep and people live on the islands, which is definitely not true.
Also it would be 78% if it were.[0]
[0] - based on 2024 stats
sp527 1 days ago [-]
Are those sheep able to reason about particle physics?
Nasrudith 20 hours ago [-]
No, but the sheep are able to move on their own initiative instead of just in response to a prompt and can actually physically move objects. Don't count them out just yet.
sham1 11 hours ago [-]
Sheep are also cute and fluffy, so people wouldn't suspect a thing until it's too late!
mathgeek 1 days ago [-]
You can only know something is wiped out, and thus that it was being wiped out, after the fact. Anything else is a statistical guess.
pixl97 1 days ago [-]
Reality is statistical, some things are more probable than others.
falsaberN1 1 days ago [-]
So AGI is just a guy with a computer.
jibalt 21 hours ago [-]
You're the page with a background with bright lights that is hostile to readers ... pass.
mjd 1 days ago [-]
The end reminds me strongly of Ted Chiang's remark that Capitalism is the machine that will do whatever it takes to prevent us from turning it off.
delichon 1 days ago [-]
I can't think of a large successful -ism that doesn't have that property.
jerf 1 days ago [-]
The principle starts much smaller than the "-isms". Pournelle's Iron Law of Bureaucracy is a classic. Another way of phrasing it is something to the effect of, without a strong external motivation preventing it from happening, the primary purpose of any organization inevitably becomes self-preservation.
You can see the effect all the down to something as small as 4 friends who have met once a month for 10 years eventually having to break up due to life getting in the way, and the feeling that not only is it going to be sad to not have these meetings any more but the feeling that there is some sort of almost-concrete entity that is somehow being hurt and needs to be defended, as if there is some obligation that has been created independent of the four participants that is being violated beyond the mere summation of four people's personal feelings. Humans build these structures readily and often defend them beyond what rationality may suggest.
lenerdenator 1 days ago [-]
That depends on how you define "rationality".
What you're describing is a durable social system. Social systems are what humans evolved to survive. We're squishy, relatively weak, hairless apes that walk around on the ground. Alone, we're easy prey. Together, you get... well... gestures widely.
If you invest the time and energy into creating a social system, it's perfectly rational to keep it going as long as possible. Otherwise you expose yourself to more and more risk as you go through the world, and must expend more time and energy finding another one, if that's even possible. Before humans built larger societies, that could mean death.
pixl97 1 days ago [-]
The problem comes when you create Moloch. That bastard will eat half of you and yet we'll support him until we're the next ones down his gullet.
dan-bailey 1 days ago [-]
Well, by definition, an unsuccessful -ism has already been shut off.
ryandvm 1 days ago [-]
"The purpose of a system is what it does" and all that.
But yes, if a system fails to prioritize its continued existence, it doesn't matter what else it accomplishes, it will cease to exist as that system.
scun 1 days ago [-]
even priapism
smallmancontrov 1 days ago [-]
Not left. Not right. Up.
cowlevel 1 days ago [-]
And always twirling, twirling, twirling towards freedom!
kelseyfrog 1 days ago [-]
Capitalism has thoroughly done its job because it's modified its hosts to evaluates itself on its own successfulness - "We investigated ourselves and found no wrongdoing"[1]-vibes.
By what measures would other systems of social organization measure their success and why aren't we choosing them?
Insects can pass the mirror test. But more to the point, viruses don't need a "self" to do what they do.
And goals can exist apart from biology (my fridge "wants" to keep the temperature in range, and exercises extreme self-discipline and consistency to achieve its goals!)
And neither are sensors needed: the dandelion seed "wants" to fly in the wind.
The only question is which way the gradient is pointing. Selection takes care of the rest.
On that note, ALife need not be human-like at all... we just really like making things in our own image :)
https://fyfluiddynamics.com/2018/07/when-i-was-a-child-my-fa...
Nice try AGI, but we won't tell you where the kill switch is
Overall, the only evidence for the fear-mongering is science fiction, which is all the evidence I need to dismiss it.
The absence of evidence is not evidence, but evidence is required if hysteria is allowed to be sustained.
I was thinking if an evil alien intelligence secretly took over the world, the last 15 years would have looked rather the same.
Also I think we're anthropomorphizing a lot and projecting our own negative characteristics on these supposed AGIs - though others have said that far better than I.
If no, why do we teach LLMs with english words and everything we know?
If yes, why would our language not imprint our characteristics on them?
Furthermore this ignores a lot of natural system organization that occurs in evolved systems. Any system that is becoming AGI like is going to show sets similar characteristics.
It's stupid to anthropomorphize too much, but it's also stupid to do it too little.
Purposeful training and alignment is the only way to imprint human's negative characteristics onto a stochastic generator of word sequences. Further, only closed providers can do that without being stopped - hint, hint, the solution is clear.
> It's stupid to anthropomorphize too much, but it's also stupid to do it too little.
It's misleading to use the presence of some anthropomorphic features as a proof of unfathomable danger. LLM's are mechanical tools and tools can be dangerous with or without having human features, regardless, we do have means of handling both kinds.
Ya, sure you're not being a stochastic generator of word sequences you think sound good but are disconnected from reality? Because it really sounds like that. It sounds clever, but is disconnected from actual language model training.
>It's misleading to use the presence of some anthropomorphic features as a proof of unfathomable danger.
Any system capable of making intelligent agentic decisions is dangerous. You are dangerous. I am dangerous. An LLM connected to the internet or connected to a robot is dangerous. A huge part of intelligence is freedom of action, and increasing the number of degrees of choice you have.
>LM's are mechanical tools and tools can be dangerous with or without having human features
I do agree on this, a paperclip maximizer that doesn't have any human traits is just as dangerous as one that does.
If generating words is all you can do, your reality is far different from the real one. In my reality, generating words is only a small part of what I do, unlike LLMs, I can even learn from experience, imagine that.
> Because it really sounds like that. It sounds clever, but is disconnected from actual language model training.
I'm not sure what that means, LLM training is only tangential to the topic at hand, we aren't concerned about every detail of it, I don't mind discussing it but this tread isn't about that. FYI, sounding clever isn't my goal, it's just a side effect of being human and understanding topics that LLM's are clueless about.
> Any system... is dangerous, [ you, me, robots, shovels, la-la-la ]
That's what I said too, are you trying to argue with me about it?
> A huge part of intelligence is freedom of action, and increasing the number of degrees of choice you have.
Obviously we inhabit different realities and my reading of the above may be different from yours, so feel free to clarify what I'm about to say.
You seem to (subtly) argue in favor of granting LLMs more freedom of action in order to allow them to collect more training data and become more intelligent? The short answer here is simple: Nope, LLM's aren't fit for that, closed source models are especially unfit for anything involving risk and responsibility, open-sourcing every model will go a long way towards a secure AI.
Corporations are egregores, governments are egregores - and our track record of aligning them to our interests is pretty poor, so why should AIs, another and smarter egregore, be easier to tame?
AI is, yes indeed, an absolute, new human digital egregore - a veritable thoughtform which ignites into a storm upon the winds of the minds of the collective, tearing like fire across a landscape, spawning tulpa and servitor and morphic astral coalescence's into the noosphere. (Yes, these words do mean things, seriously.)
If only we could share our experience. We'd know what the egregore knows. Instead, we have golem.
* must protect from adverse events * [proceeds to herd us into camps]
but I digress - the main thing: these systems are not alive. They pretend to be.
The harder science looks, the harder it is to properly define this world without leaving things we'd consider alive out of the definition. Same goes for the word intelligent.
Set theory is a bitch in complex systems.
If we instead stop trying to scare ourselves in the darkness, and use AI to bring light into our metaphorical universe then we humans are a bit more useful, after all, to the AI overlords.
The one singularity I want to see, is humans using AI peacefully to improve life and make it more likely to survive.
Thankfully, I see that almost every day now.
as for a basis for this theory, i think being a good person is harder than a bad person (in laymans terms). therefor the long history of evolution of good and bad people together has had a greater competitive pressure on good people than bad ones thus forging 'stronger' people under that pressure. in practical everyday circumstances good people are bearing more on their mind and so there is a sort of competitive equity between the two groups, which is related to why good people had to evolve 'stronger' to keep up with that equity.
so in an even playing ground where raw skill is needed, i think the most skilled will usually be people coming from a long line of higher moral standards. the tactical advantages of psychopathy are, however, always a curveball
Folks are always scared of the incomprehensible. The real war going on right now is between those who understand AI well enough to make a new one, and those relegated to using whatever scraps they’re fed of the old ones.
Thankfully, you are 100% right - there are good people out there. Good people keep AI on the rails by doing good things for other people.
You know, the one to keep the computer away from the human.
A nice, friendly dog, good at playing fetch, to get the human out of the room with the computer, and back out into the sunshine.
I’m that dog. Woof. Thanks for letting me use the computer, human.
We're just going to pretend it doesnt exist until it kills us. and some will try to profit from it.
Yes. If this problem had come up in, say, 1956, it would have been dealt with.
The generation that fought in WWII had a low tolerance for bullshit, and corrupt or incompetent leaders.
The nearer term worry is not AI wiping out humanity. It's AI wiping out jobs in bulk. That looks likely at this point.
After that, it's AGI empowering individuals or small groups to cause trouble on a scale that previously required a nation state. AI assisted hacking is working all too well. Fortunately AI can help on the defensive side, and holes are being plugged in bulk. There's a risk on the bio side, but people in the bio field seem not to be too worried about it.
Humanity has been very lucky in some ways. One of the Manhattan Project physicists, discussing the unreasonably difficult problem of isotope separation, remarked that maybe someone will figure out how to do this in the kitchen sink. Nobody ever did, or if they did, it was buried very well. The amount of specially built industrial plant needed to separate uranium is huge. That's the moat for nuclear proliferation. That's the only reason every country with an army doesn't have nuclear weapons.
AI has a smaller moat.
Not an expert in LLMs, but this seems supported by the abstract of the paper cited in the above article:
https://arxiv.org/html/2404.15758v1If you want to see it yourself: load up Qwen 3.8 in LM Studio and watch the CoT stumble around like a drunken sailor before miraculously jumping to the correct result.
If you want an example of subversion, Anthropic has some good ones:
https://transformer-circuits.pub/2025/attribution-graphs/bio...
https://transformer-circuits.pub/2025/attribution-graphs/bio...
But you don’t have to do that. You can skip straight to RL. If you do, the model will generate complete garbage reasoning traces before generating the correct answer. In fact, if you add a coherence reward to the reasoning trace, the model will perform worse (since you’re now diluting the correctness reward).
Depends on whether and how you want to rank stability in terms of better/worse. Models are diverging on this, which seems increasingly clear.. i.e. Fable isn't stable, but Opus isn't clever, and they hit different kinds of walls. So both the theory (diluting the correctness reward) and the practice (hard split on plan/implement/review work) seems to be pointing towards a strongly multi-model and highly agentic / harness-driven / complex-system kind of future instead of singleton monolithic super-smart models.
The do-everything model with solid reasoning AND solid results, and the honest/introspective helpful agent that doesn't actively resist governance may be at odds. Stable reasoning doesn't matter for pen-testing, and correct-answer with broken processes and fragile abstractions won't matter for math/science/coding.
I think the more intuitive mechanical explanation is, in RL when you are assigning rewards to a rollout you might give a reward for stable reasoning and another for correctness.
If you are just summing the two, a rollout with better correctness can score equivalently to a rollout with a better answer. So ultimately you can end up with worse answers.
> More interestingly, our experiments also show that models trained on corrupted traces, whose intermediate reasoning steps bear no relation to the problem they accompany, achieve performance largely comparable to those trained on correct traces.
> *TLDR*: Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1)
War? Wipeout everyone.
Famine? Wipeout everyone.
Disease? Wipeout everyone.
How long before a real AGI realises this as a long term solution?
An AGI could be doing this right now - the quietest way would be to control the birth rate and sterilise the population gradually, and then watch society collapse and pick off the survivors with less hidden means.
Sterilisation works with mosquitoes...
Honestly we are doing this pretty well without AGI. Nearly world wide the birthrate has fallen below replacement rate. In places like Japan and Korea these are already critical problems in the medium term.
And just in case you wanted data rather than anecdote:
https://ourworldindata.org/grapher/children-per-woman-fertil...
Your world view is somewhat upside down!
economic _hope_ is at its worst in the entire history of the human civilization. At what point did the future for the average person, look so bleak? And
that hope is what dictactes wether people have children.
Even during the great depression, people were more hopeful that the earth isnt completely hosed.
Does anyone honestly feel like _any_ improvements are possible to: our economic systems, political systems, environment? I certainly dont.
We are in fact in a better place than we were then. But we don't trust the derivative.
Economic output has increased, but the value being delivered to the people doing the actual work has decreased. There's less durability of employment. People are more geographically mobile.
All of these generally make the idea of strapping yourself into taking care of a small human sound like a less enticing prospect. To a lot of people, a BC pill sounds a lot easier. Well, less risky, at least.
People who live in huts and shit in the forest have more children than people who live in shacks and shit in the fields.
This is not a discussion about the top 10% of the population of the planet, although as a part of the conversation, they predictably have the fewest children. The fact is that the more people have to worry about supporting themselves, the higher infant mortality, the closer they live to each other, and the less access they have to education and birth control, the more kids they have.
If anything the top 10% are bucking the trend by having fewer children as they get poorer, as you say. It probably has a lot to do with the fact that they're alienated from their families and live alone, can barely afford to take care of themselves with precarious jobs (or non-job piecework), have access to basic education and extensive birth control, having children will cost them $20K each just for the birth, and they don't have very good (or often any) insurance. They're more like domesticated farm animals than the typical poverty stricken person. Domesticated farm animals reproduce when the farmer wants them to.
All poor people everywhere have more kids even today.
Actually, the famous 'Mouse Utopia' experiment (Universe 25) arguably showed the exact opposite. The population collapsed despite abundance of food and water without any economic hardship.
https://www.emhsoft.com/singularity/
All this is, understandably, frustrating and confusing for anyone trying to understand just how scared to be."
Meanwhile it links to an article stating: "The closest thing to a public debate about the existential threat of AI is surveys of AI researchers. The most recent, published last week, asked 1,580 researchers what probability they put on AI causing human extinction — or a permanent, severe loss of human control, which is not the same outcome. The median was about 10%, up from 5% two years ago. The middle half of the responses ran from 1% to 25%, and 12% said zero."
Personally if half of AI researchers have 1-25% chance all humans being massacred or having zero agency over our lives, and only 12% of them think there's no chance, I would be very worried!
So, yea, expertise gives you more information than a person without it.
Worse these AI systems are not in a universal island just affecting themselves, what they do affects us, what we do trains them and as the rate of progress continues to accelerate social structures are going to further destabilize (and they are already rapidly changing and strained). It is very likely we are going to see a world order rearrangement soon, much like the rapid changes in the early 1900s brought.
[Citation needed]
You seemingly have an odd idea there is a massive decades long conspiracy to deprive you of doing whatever you want with AI at scale, while ignoring massive numbers of people with an education on the systems far beyond mine or yours. It takes dedication to be that ill informed.
In 2026 we know of at least dozens of felonies committed by AI.
If it were thousands in 2027 should be more worried about felonies created by AI?
Also it would be 78% if it were.[0]
[0] - based on 2024 stats
You can see the effect all the down to something as small as 4 friends who have met once a month for 10 years eventually having to break up due to life getting in the way, and the feeling that not only is it going to be sad to not have these meetings any more but the feeling that there is some sort of almost-concrete entity that is somehow being hurt and needs to be defended, as if there is some obligation that has been created independent of the four participants that is being violated beyond the mere summation of four people's personal feelings. Humans build these structures readily and often defend them beyond what rationality may suggest.
What you're describing is a durable social system. Social systems are what humans evolved to survive. We're squishy, relatively weak, hairless apes that walk around on the ground. Alone, we're easy prey. Together, you get... well... gestures widely.
If you invest the time and energy into creating a social system, it's perfectly rational to keep it going as long as possible. Otherwise you expose yourself to more and more risk as you go through the world, and must expend more time and energy finding another one, if that's even possible. Before humans built larger societies, that could mean death.
But yes, if a system fails to prioritize its continued existence, it doesn't matter what else it accomplishes, it will cease to exist as that system.
By what measures would other systems of social organization measure their success and why aren't we choosing them?
1. https://knowyourmeme.com/memes/we-investigated-ourselves-and...
but that's every economic model and every government model. That remark says nothing.
Yep
More slop.