General models can run robots. They are very bad at it. It is much more efficient to have a stack of models with 'dumber' but faster models taking care of lower level functions and higher level ones that are much slower taking care of high level planning. Consciousness isn't super fast, but it is very smart. Your 'lizard brain' (to use an overstretched metaphor) and reflexive system are relatively stupid but far faster and condense a whole raft of inputs to something your consciousness can deal with. For robots it isn't all that different. Sensors and actuators require millisecond level responses, that's something you're not going to get out of an LLM on the other side of a communications link.
mrtksn 1 days ago [-]
What models actually run? Do they generate the PWM signals/their widths to drive a servo or something? Or is it more like "joystick commands" generator?
jacquesm 1 days ago [-]
More like high level motion controller, so 'servo 23 1700 us' or for better grade servos 'servo 23 pos 145400', possibly concurrent with motion commands to other actuators and possibly synchronized, possibly using other kinds of protocols (G-codes).
Motion controllers are pretty impressive these days. PWM generation is too time critical to leave to high level processors, usually this is done in hardware if you want it to be guaranteed glitch free but there are also 'bit banging' servo controllers out there.
mrtksn 1 days ago [-]
Interesting, so how does the inference work here? are they aiming for a fixed length intervals like 10 times/second generating the positions or is it more like reactive, i.e when the request to hold a cup from a higher level comes then this is treated as a promt, the sensors are the context and when the inference finished the pulses are sent according to the output?
jacquesm 19 hours ago [-]
That's roughly how I've set up one of my environments. But how long inference takes is highly dependent on which sensors are in play at that moment.
ajw287 17 hours ago [-]
"Here I am, brain the size of a planet, and they ask me to pick up a piece of paper." Douglas Adams (Marvin)
jacquesm 16 hours ago [-]
Marvin is priceless.
'"Can I open this door" you ask?'
gopalv 1 days ago [-]
> Sensors and actuators require millisecond level responses, that's something you're not going to get out of an LLM on the other side of a communications link.
We've gotten to slightly a frame behind for the models I'm using to play video games - 36ms to get the next action.
The idea is probably not have a frontier model doing this, but to make a frontier model build these out of its own brain (well, directly edit the weights instead of RL tuning).
My team did call these sort of slices of a main brain a "Reflex agent" in similar fashion, since they do not reason only respond.
dwattttt 1 days ago [-]
The faster you are, the more systems you're able to control directly. At ~30ms per "choice"/output, you can choose what lower level motors/servos/systems drive towards, but you can't generate the signals that drive them. You're looking at 1kHz and maybe higher there.
baq 1 days ago [-]
> General models can run robots. They are very bad at it.
Reminder frontier labs are 6 months ahead of public models internally, if not more
jacquesm 1 days ago [-]
That does not make them orders of magnitude faster.
euroderf 6 hours ago [-]
Has anyone tried to plot a "Moore's Law" for A.I. ?
1718627440 12 hours ago [-]
/dev/urandom can run robots. Granted most only for a few milliseconds, but given enough controlled robots, some will make it out of the door.
raincole 1 days ago [-]
General models can {insert something general models will be very good at}. They are very bad at it.
Every single time.
cobychapple 1 days ago [-]
nothing says reflexes like getting a 429 rate limit halfway down the stairs.
yewenjie 1 days ago [-]
The Bitter Lesson comes for everything.
glalonde 1 days ago [-]
the bitter lesson assumes compute is cheap and abundant, which is not really true in most robotics contexts. Unless your datacenter is right next to the robot, and you are streaming real time sensors to it.
robotresearcher 19 hours ago [-]
Even then it also needs to make economic sense. Your robot has to do something valuable to pay for the tokens.
_diyar 1 days ago [-]
The Bitter Lesson comes for everything eventually.
lukan 1 days ago [-]
"which is not really true in most robotics contexts. Unless your datacenter is right next to the robot"
Most robots do run where there is internet aka a datacenter right next to them. Otherwise sure, a fable class model is not the right way for making a robot walk forward. But to analyze periodically the state of it - maybe.
adamnemecek 1 days ago [-]
Also that compute is fast enough to make split second decisions.
woopsn 1 days ago [-]
Reads as something different. If you're building robots you need a robotics team, regardless. And if your general model blows specialized ones out of the water, send someone to CoRL.
1 days ago [-]
AndrewKemendo 1 days ago [-]
I’m flabbergasted that anyone disputes this anymore.
jvanderbot 1 days ago [-]
Let me dispel one myth: The bitter lesson can be true, and you can still want a bespoke, customized solution.
If it is true that eventually a general approach that scales with compute/data will supersede a bespoke expert solution, it is not necessarily true that today's expert solution should be thrown away at the first sign of a data-driven model.
It says nothing about what you should use today to build things, it says everything about how you should do research that lasts.
If you're building for billions of users in 10 years, sure go with whatever you think is the bitterest.
But if you're building for right now, you probably mashup whatever works best. That might still be a PID loop, EKF, and Homographies sometimes. But when you hit the frontier of what "works", then you have to reach for new things anyway.
Chesterton's baby/bathwater
WRT humanoids - I am making no statements.
janalsncm 1 days ago [-]
As an analogy, it is currently possible to cross the country in 6 hours.
That fact does not imply that a person in Boston in the year 1850 should wait until commercial flight was available to travel to California.
So the fact that we will eventually have simpler/more efficient algorithms doesn’t tell us anything about what to do today.
adrianmonk 1 days ago [-]
Or, if you want to make a plastic phone case, 3D printing can absolutely do that for you. But if you look for a way to make thousands of high-quality phone cases quickly and cheaply, you'll realize that injection molding hasn't been made obsolete.
AndrewKemendo 1 days ago [-]
>It says nothing about what you should use today to build things, it says everything about how you should do research that lasts.
Right! So who are you arguing with lol?
qlte 1 days ago [-]
"Bitter lesson" gets pattern matched as a stock answer to anything vaguely resembling optimization/compute trends these days.
Like, news about internal reorgs at one company that could plausibly be done for any number of boring, bean-counting reasons.
From the article it seems more likely OpenAI just isn't prioritizing robotics anymore. Which seems to fit better than abandoning their booth at a robotics expo because "robotics = solved".
The change came recently enough that OpenAI had already registered a booth at CoRL, a top academic robotics conference in Austin, this week. Though the conference app and floor plan marked OpenAI’s spot, when Fortune visited booth #208, it sat empty with OpenAI’s logo nowhere to be seen.
ambicapter 1 days ago [-]
Everyone very usefully saying "bitter lesson lol" in the comments.
ainch 23 hours ago [-]
Sutton himself still thinks the current methods are insufficient. The question is not whether the Bitter Lesson is true, but whether we have algorithms that scale sufficiently with compute to warrant no further research.
Yajirobe 1 days ago [-]
DeepSeek disputed it
scottLobster 1 days ago [-]
The Bitterer Bitter Lesson is that there are no exponentials, only S-Curves.
You can render graphics with the fastest CPU you can find if you like, I'll keep my dedicated GPU
AndrewKemendo 1 days ago [-]
They don’t conflict
winfredJa 1 days ago [-]
This looks like general cost cutting to boost some numbers before IPO.
seanmcdirmid 1 days ago [-]
They didn't fire or lay them off, they just disbanded the team and moved the researchers to other teams, so how is it cutting costs?
spamizbad 1 days ago [-]
Robotics is more capital intensive than software engineering.
ImJasonH 1 days ago [-]
More wood behind fewer arrows
qlte 1 days ago [-]
Moving resources from cost center to profit center.
lavendermon 1 days ago [-]
Not necessarily cost cutting, just streamlining and focusing on a few core products that can make money. ChatGPT has leveraged its writing style into a product that many consumers choose for their basic AI needs. The company is using information picked up in those chats to personalize the outputs, build trust, and most importantly raise switching costs as the users won't want to lose their personalized helper and confidant.
I'm already seeing some of my family members use ChatGPT in this way, even going so far as enjoying some level of companionship during research and decisions in their lives. I believe this product will have fantastic pricing power and can make the ChatGPT IPO look very attractive.
alsetmusic 1 days ago [-]
> I'm already seeing some of my family members use ChatGPT in this way, even going so far as enjoying some level of companionship during research and decisions in their lives.
I’d also create a throwaway account if I was going to say something this cringe.
graemep 1 days ago [-]
its cringe, but its true.
I suspect a major use of AI will be providing companionship to the lonely, and emotional support, and lots of similar things. Look at r/myboyfriendisai Look at reports of people using it as therapy.
The go and read The Machine Stops by EM Forster.
AspireOne 1 days ago [-]
I am 'the lonely', and I indeed use ChatGPT this way.
hdndjsbbs 1 days ago [-]
What exactly are people going to do in the future? Not work and not talk to each other? AI maxis seem to believe interpersonal interaction is a thing of the past
graemep 17 hours ago [-]
People already interact far less than they used to.
A lot of community and family interaction has been replaced by professional interaction - from childcare to therapists. That could be automated.
Work is already somewhat down this road. Reading an AI summary of an AI written email, for example. People spend more time on social media than in third spaces and increasingly post AI written posts there.
MisterMunchkin 15 hours ago [-]
What do they do at the minute?
(They sit at home gooning to tikslop)
chis 1 days ago [-]
lmao
eikenberry 1 days ago [-]
So they gave up on non-cloud connected robotics then? Or was that ever in their scope?
janalsncm 1 days ago [-]
Counterintuitively, it’s a lot easier to train a big model and distill it down to a small one than to train the small one from scratch. So never say never.
throwaway63467 1 days ago [-]
Who says you can’t run an LLM inside a robot if you have enough energy and hardware?
torginus 1 days ago [-]
If those models can run locally and according to the article they're the same ones that do everything else - that means their models are quite small and cheap, which would be suprising. Even a machine running a 30B quantized LLM (which is generally the lower bound of usefulness) at meaningful speeds costs tens of thousands, and that's just the silicon, never mind building an actual robot around it.
jacquesm 1 days ago [-]
It isn't even optional. Anything with a motor should have sufficient executive authority on board to decide on how to safely power down in case of trouble. Loss of communications is a major impediment to deploying robotics to the point that I think it should be outlawed to have robots that do not have all of their thinking done locally.
But I suspect that this will be one of those regulations written in blood.
1 days ago [-]
lovich 1 days ago [-]
At the bare minimum there should
be a local independent system that safely shuts down the robot if it doesn’t get heartbeats from their control network
eikenberry 1 days ago [-]
No one did. I was saying that those models would be small, dedicated models like those possibly developed by a robotics team. OpenAI's general purpose models are not small, dedicated models.
ph4rsikal 1 days ago [-]
[dead]
deadbabe 1 days ago [-]
Is there a primer on how to actually use LLMs to control robots? Seems very token inefficient if you have to stream commands at 60fps.
tracerbulletx 1 days ago [-]
Robotics are generally layered systems with lower latency/realtime systems running literally on the servos and other embedded hardware that use servo feedback and direct sensor access to operate, then building up complexity with tiers of increasing policy, usually all capable of working without the higher order layers which can inject desired states into lower order rings.
The LLM's are basically used for the big picture. What is the situation. What is the goal and how to achieve it. Not direct control of all the servos.
mr_toad 1 days ago [-]
The headline is somewhat misleading, as is tradition.
almostdigital 1 days ago [-]
I think the idea is to just have the LLMs write the control systems and RL on that
dwattttt 1 days ago [-]
Sure hope they're not writing the safety cutoffs as well. Some of those systems do not take kindly to bad control inputs, and determining if a controlled system is stable can be absurdly hard.
ares623 1 days ago [-]
Depends on what your goals are. If your goal is to give investors erections, then re-using your existing trillion dollar model is efficient.
photochemsyn 1 days ago [-]
The termite mound model for coordinating robots using a harness-type LLM system all encapsulated within a non-LLM deterministic controller that rejects banned actions the LLM proposes, with individual termites running their own small efficient stacks and following commands from the LLM hive mind, that’s one way to do it. The main idea behind such architecture is that the controller prevents banned actions (like hacking HuggingFace) by the hive mind LLM and its swarm of agentic termites. Once you accept that alignment can always be internally subverted, it’s the only plausible safe architecture.
andriy_koval 1 days ago [-]
they have decision models now
Tadpole9181 1 days ago [-]
Saved the NVIDIA paper below for reading material, but if I had to guess just as a SWE how I would do it: use hierarchal thinking.
At the top sits a SOTA model capable of seeing snapshots of what's happening and directing intent. Then smaller, more reactive models translate the intent down into actions, before coming to a purely deterministic layer that handles inverse kinematics.
Sorta like how I control my body. I usually don't think about my fingers going to every key as I type - I know what I want to type, then that signal goes to some autonomous "using a keyboard" system, which invokes my muscle memory, which actually moves my muscles.
And if I get burned while I do something, my actual thinking brain doesn't really have time to process it before I'm already moving away; because those smaller systems have much faster - but more rudimentary - feedback loops.
jeffbee 1 days ago [-]
I wonder if this means anything about OpenAI's robot research office in Richmond, CA.
metalliqaz 1 days ago [-]
money is tight
brink 1 days ago [-]
It's ironic that the people working to make others disposable were in fact, disposable.
AI is eating AI now. The value of code has collapsed, while the value of the software industry overall has not increased. Could AI be the snake that eats its own tail in the long-term?
c1ccccc1 1 days ago [-]
According to the article at least, all these people still have jobs at OpenAI.
brink 1 days ago [-]
Thanks for pointing that out.
pixl97 1 days ago [-]
"Make a general intelligence, get surprised when it generalized "
Now your part of snake eating it's own tail makes little sense, AI isn't going away, only bad business models.
NortySpock 1 days ago [-]
Open AI realizes robots with actual hardware are hard, and expensive.
dude250711 1 days ago [-]
"long-term vision" - not even mid-term? I hope Anthropic knows better...
mequelkramer 1 days ago [-]
[flagged]
elvisgodevelop 22 hours ago [-]
[flagged]
hulitu 16 hours ago [-]
> OpenAI disbands its robotics team as its general models start running robots
So the robotic team operated the Exchange Mail servers at Microsoft. Now it will be Copilot. That makes sense. /s
Rendered at 00:24:57 GMT+0000 (Coordinated Universal Time) with Vercel.
Motion controllers are pretty impressive these days. PWM generation is too time critical to leave to high level processors, usually this is done in hardware if you want it to be guaranteed glitch free but there are also 'bit banging' servo controllers out there.
'"Can I open this door" you ask?'
We've gotten to slightly a frame behind for the models I'm using to play video games - 36ms to get the next action.
The idea is probably not have a frontier model doing this, but to make a frontier model build these out of its own brain (well, directly edit the weights instead of RL tuning).
My team did call these sort of slices of a main brain a "Reflex agent" in similar fashion, since they do not reason only respond.
Reminder frontier labs are 6 months ahead of public models internally, if not more
Every single time.
Most robots do run where there is internet aka a datacenter right next to them. Otherwise sure, a fable class model is not the right way for making a robot walk forward. But to analyze periodically the state of it - maybe.
If it is true that eventually a general approach that scales with compute/data will supersede a bespoke expert solution, it is not necessarily true that today's expert solution should be thrown away at the first sign of a data-driven model.
It says nothing about what you should use today to build things, it says everything about how you should do research that lasts.
If you're building for billions of users in 10 years, sure go with whatever you think is the bitterest.
But if you're building for right now, you probably mashup whatever works best. That might still be a PID loop, EKF, and Homographies sometimes. But when you hit the frontier of what "works", then you have to reach for new things anyway.
Chesterton's baby/bathwater
WRT humanoids - I am making no statements.
That fact does not imply that a person in Boston in the year 1850 should wait until commercial flight was available to travel to California.
So the fact that we will eventually have simpler/more efficient algorithms doesn’t tell us anything about what to do today.
Right! So who are you arguing with lol?
Like, news about internal reorgs at one company that could plausibly be done for any number of boring, bean-counting reasons.
From the article it seems more likely OpenAI just isn't prioritizing robotics anymore. Which seems to fit better than abandoning their booth at a robotics expo because "robotics = solved".
You can render graphics with the fastest CPU you can find if you like, I'll keep my dedicated GPU
I'm already seeing some of my family members use ChatGPT in this way, even going so far as enjoying some level of companionship during research and decisions in their lives. I believe this product will have fantastic pricing power and can make the ChatGPT IPO look very attractive.
I’d also create a throwaway account if I was going to say something this cringe.
I suspect a major use of AI will be providing companionship to the lonely, and emotional support, and lots of similar things. Look at r/myboyfriendisai Look at reports of people using it as therapy.
The go and read The Machine Stops by EM Forster.
A lot of community and family interaction has been replaced by professional interaction - from childcare to therapists. That could be automated.
Work is already somewhat down this road. Reading an AI summary of an AI written email, for example. People spend more time on social media than in third spaces and increasingly post AI written posts there.
(They sit at home gooning to tikslop)
But I suspect that this will be one of those regulations written in blood.
At the top sits a SOTA model capable of seeing snapshots of what's happening and directing intent. Then smaller, more reactive models translate the intent down into actions, before coming to a purely deterministic layer that handles inverse kinematics.
Sorta like how I control my body. I usually don't think about my fingers going to every key as I type - I know what I want to type, then that signal goes to some autonomous "using a keyboard" system, which invokes my muscle memory, which actually moves my muscles.
And if I get burned while I do something, my actual thinking brain doesn't really have time to process it before I'm already moving away; because those smaller systems have much faster - but more rudimentary - feedback loops.
AI is eating AI now. The value of code has collapsed, while the value of the software industry overall has not increased. Could AI be the snake that eats its own tail in the long-term?
Now your part of snake eating it's own tail makes little sense, AI isn't going away, only bad business models.
So the robotic team operated the Exchange Mail servers at Microsoft. Now it will be Copilot. That makes sense. /s