Real biological operant behavior isn't exactly trial and error learning.
Many factors shape and guide initial responses.
What I've noticed in some descriptions of models is the use of optimization for reinforcement to shape responses. In real organisms behavior may be controlled by short or long term outcomes, and may oscillate between this "optimization" based on schedules. This produces variability in the trials which can adjust behavior. Are we seeing these reinforcement models do this?
herodoturtle 14 hours ago [-]
I found this comment/question deeply intriguing.
I’m no expert at this and was wondering what you meant by the following:
> In real organisms behavior may be controlled by short or long term outcomes, and may oscillate between this "optimization" based on schedules
Could you perhaps provide an example that would help me understand what you mean?
Thanks for the insightful comment either way.
ainch 10 hours ago [-]
There is a field of hierarchical RL in which the optimisation occurs over a range of time scales/abstraction. But I'm not aware of much practical success for these approaches so far.
programjames 19 hours ago [-]
I skimmed through the book, and it's lacking the information theory foundations. For example, "trust region methods" come from maximizing the policy's relative entropy (to a reference policy) under a tournament system where high-scoring agents are exponentially likely to survive. In general, a reward is the negative bits it costs an environment to propagate an agent (multiplied by some temperature).
ainch 10 hours ago [-]
Do you have a good source on this information theory framing? I don't remember it being covered in Sutton & Barto.
porridgeraisin 6 hours ago [-]
It's just another way to frame it. It's as foundational as the many other ways to frame it. I'm not aware of any major insight you get specifically from this framing. Is there one?
laurensr 7 hours ago [-]
This reminds me of the Little Book of Calm, discussed extensively in the Black Books TV series.
wpm 6 hours ago [-]
Thank you for reminding me to rewatch Black Books.
janalsncm 12 hours ago [-]
I wonder what Sutton thinks about some of the more recent innovations in RL like GRPO. In some ways it’s new, in other ways it’s an echo of RLOO.
porridgeraisin 6 hours ago [-]
GRPO is policy gradient/PPO with your value function baseline monte carlo estimated using k rollouts. The only new thing is finding out it works well with binary rewards and LLM policies.
janalsncm 2 hours ago [-]
It is a huge improvement to PPO because you don’t need a separate critic model which cuts memory costs in half and stabilizes training.
definitely the latter, it is even referenced in the foreword:
> Its goal is not to be exhaustive, but rather minimalist
and easy to read. For this reason, it follows the format
of The Little Book of Deep Learning [Fleuret 2023]. Its
tone, however, is closer to that of a blog post, as the
book is built around a single narrative thread. Its
structure broadly follows that of Sutton and Barto’s
Reinforcement Learning: An Introduction [Sutton et al.
2018], which remains the canonical reference on the
subject.
tejtm 13 hours ago [-]
The Little Schemer, The Little Typer, The Little Reasoner, The Little Proover
The Little MLer ...
It has been going on for a while in Lispy land
Exoristos 20 hours ago [-]
No, it's _obviously_ homage to the Little Liddel[0].
Many factors shape and guide initial responses.
What I've noticed in some descriptions of models is the use of optimization for reinforcement to shape responses. In real organisms behavior may be controlled by short or long term outcomes, and may oscillate between this "optimization" based on schedules. This produces variability in the trials which can adjust behavior. Are we seeing these reinforcement models do this?
I’m no expert at this and was wondering what you meant by the following:
> In real organisms behavior may be controlled by short or long term outcomes, and may oscillate between this "optimization" based on schedules
Could you perhaps provide an example that would help me understand what you mean?
Thanks for the insightful comment either way.
Often referred to as "The Little Book".
https://fleuret.org/francois/lbdl.html
> Its goal is not to be exhaustive, but rather minimalist and easy to read. For this reason, it follows the format of The Little Book of Deep Learning [Fleuret 2023]. Its tone, however, is closer to that of a blog post, as the book is built around a single narrative thread. Its structure broadly follows that of Sutton and Barto’s Reinforcement Learning: An Introduction [Sutton et al. 2018], which remains the canonical reference on the subject.
It has been going on for a while in Lispy land
0. https://archive.org/details/lexiconabridgedf00liddrich
https://imgur.com/zMTEE