Ha, interesting. I wasn't aware of Sutton's blog post, but if I might make a shameless plug, we demonstrated [1] exactly this problem (see section 4.4.3), and how multi-step world models (using diffusion models as the substrate) could be one potential answer.
Since then, I have come to like temporally-abstract models more and more. Rolling out in time -- either step-by-step or many steps at once -- suffers from the tyranny of the specific. For long horizon planning with agents, I care (often only approximately) about where I can end up, and seldom about exactly when I end up there. Successor features, GVFs, Forward-Backward representations, and the like seem like they have an elegant approach for structuring thinking at a "high level", instead of generating exponentially large search trees by rolling out microscopic world models.
Imagine I want to attend a conference in a different country. Google maps might give turn by turn navigation but that is an overwhelming and largely irrelevant mess of details for most planning purposes. Eg: all I might want to know is the different flight legs and the fact that the journey takes 15-18 hours, and not all the turns and traffic lights to get from home to the airport.
I want a zoomed out picture, and to be able to fill in detail hierarchically, on demand.
Instead, one-step models give you the full high-res local structure of the graph that would have to search through (with too many states and edges).
mxwsn 1 days ago [-]
This is the same reasoning behind why Yann Lecun thought test-time scaling would not work for LLMs: compounding error.
Instead, the more tokens LLMs use, the better their performance on many tasks. LLMs can self-correct, evidenced by the power of getting models to question themselves by emitting "Wait," in S1. https://arxiv.org/abs/2501.19393
rf15 1 days ago [-]
You wouldn't believe the amount of reasoning I saw these past few months that was correct until the stochastic parrot decided that a "wait" token should now be used and everything steered off a cliff.
tipsytoad 1 days ago [-]
Yeah came here to comment exactly this. And this is generally why I dislike/avoid this type of first principle analysis: it can make very convincing arguments that are just totally wrong due to some misleading assumption
fny 1 days ago [-]
I'm not sure I follow what one step means exactly. Aren't all models some f(x) = y? Is the suggestion instead that we should be doing f(x) = g(h(x)) = y?
What would the difference be?
smokedetector1 19 hours ago [-]
The fallacy is that f(t+N) can be obtained by iterating f(t+1) N times. This is the “step”
thornewolf 22 hours ago [-]
just read this when reviewing OpenAI's "spinning up" documentation as it was linked there!
gnabgib 1 days ago [-]
(2024)
kiaansaraiya 3 hours ago [-]
[dead]
nttylock 1 days ago [-]
[flagged]
Rendered at 23:11:29 GMT+0000 (Coordinated Universal Time) with Vercel.
Since then, I have come to like temporally-abstract models more and more. Rolling out in time -- either step-by-step or many steps at once -- suffers from the tyranny of the specific. For long horizon planning with agents, I care (often only approximately) about where I can end up, and seldom about exactly when I end up there. Successor features, GVFs, Forward-Backward representations, and the like seem like they have an elegant approach for structuring thinking at a "high level", instead of generating exponentially large search trees by rolling out microscopic world models.
[1] https://arxiv.org/abs/2410.05364 (funnily, from around the same time / few months after Sutton's blog post)
I want a zoomed out picture, and to be able to fill in detail hierarchically, on demand. Instead, one-step models give you the full high-res local structure of the graph that would have to search through (with too many states and edges).
Instead, the more tokens LLMs use, the better their performance on many tasks. LLMs can self-correct, evidenced by the power of getting models to question themselves by emitting "Wait," in S1. https://arxiv.org/abs/2501.19393
What would the difference be?