Pretty insane that you can get this close to the real thing this way.
Rocket league is one of my favourite games, and I'm pretty decent at it (rank champion 1). I kinda felt like my controller was a bit broken when playing this, a lot of commands were just ignored, and forget doing stuff like speed flips. But I did feel like was controlling the car, and everything about the game looked very much like the real thing. Ball movement was on point, I didn't notice any weird bounces or anything.
The lack of opponents pulling triple flip resets and double-tapping musty's (musties?) was the most notable difference from the real thing
jorl17 17 hours ago [-]
This was a much better experience than I expected. Rather unbelievable!
Side-effect of the data: clearly the model is better than I normally am at playing, as it spontaneously did several things I had not told it to do and wouldn't really know how to do (at least not with a keyboard).
Really remarkable, congrats!
MasterScrat 16 hours ago [-]
Hey all, happy to see this here! This was a colab between General Intuition (that I’m part of), Kyutai and Epic Games.
You can read plenty of details in the blog post and tech report but the TLDR is that we trained a multiplayer world model on 10k hours of Rocket League data. We optimized it to be playable at 20fps on a single GPU.
So what you see in the demo is fully generated: there’s no graphics or physics engine. Instead it’s a 5b neural network that takes actions in and gives pixels out.
pvillano 15 hours ago [-]
Could a network be trained to transform physics state directly into the latent state and back?
Having a direct transformation would enable some interesting experiments.
How is the latent state different when everything else stays the same, but you change one physics value, like player one velocity? Is there a cyclical pattern of activation that correlates strongly with the seconds digit of the clock? Can you decode the latent state, give players full boost, and then re-encode it for infinite boost, without losing continuity?
Edit: There sure are a lot of papers on interpretability.
MasterScrat 12 hours ago [-]
Would be a great idea to see how much we could manipulate the latent space and whether it has some internal structure w.r.t the physical state. I guess the only unknown is how the world model would show robustness to latent states that are transformed through this network
sliding-penguin 6 hours ago [-]
Very cool, and publishing a slice of the dataset and all of the training code is fantastic, but if reproducing the model and the video representation codec is encouraged, why not open source the models or at least some variant of them?
I'd be interested in seeing if fine-tunes that include human gameplay data would be possible.
pizzathyme 15 hours ago [-]
Tim Sweeney’s interviews on the uses of GenAI for game development have been some of the best takes I’ve heard. He’s mentioned how GenAI is great at filling in the gaps or treating assets, but no world simulation means no deep persistence or authoring for a whole new unique game world.
What is the conversation like within Epic now? Is this still the view? What is the future for simulations like this?
RugnirViking 7 hours ago [-]
The demo button, and most of the features mentioned, dont seem to work for me, on edge or chrome. Project sounds really interesting, so I wish I could try!
in-silico 15 hours ago [-]
I feel like the data should have been generated by a much less predictable policy.
It often feels like the model is ignoring my inputs and just doing what it would expect the bot to do (which is unsurprising if the model could predict what would happen next during training without paying attention to the inputs)
7 hours ago [-]
bschwindHN 16 hours ago [-]
Where is the option to call all of my tm8s trash? That's an essential part of the experience!
MitziMoto 15 hours ago [-]
What a save!
What a save!
What a save!
(Chat disabled for 3s)
coip 13 hours ago [-]
Calculated.
twright0 11 hours ago [-]
Sorry! Sorry! $#@%! Sorry!
skibz 11 hours ago [-]
Nice shot!
superkuh 16 hours ago [-]
It feels like playing on a very slow computer. Except that sometimes it just randomly decides you pressed the flip button. Really impressive.
avaer 16 hours ago [-]
If the data and code is all there, why not release the 5B weights?
Harsh_Dalal 11 hours ago [-]
[flagged]
nullsanity 14 hours ago [-]
[dead]
Rendered at 17:58:33 GMT+0000 (Coordinated Universal Time) with Vercel.
Rocket league is one of my favourite games, and I'm pretty decent at it (rank champion 1). I kinda felt like my controller was a bit broken when playing this, a lot of commands were just ignored, and forget doing stuff like speed flips. But I did feel like was controlling the car, and everything about the game looked very much like the real thing. Ball movement was on point, I didn't notice any weird bounces or anything.
The lack of opponents pulling triple flip resets and double-tapping musty's (musties?) was the most notable difference from the real thing
Side-effect of the data: clearly the model is better than I normally am at playing, as it spontaneously did several things I had not told it to do and wouldn't really know how to do (at least not with a keyboard).
Really remarkable, congrats!
You can read plenty of details in the blog post and tech report but the TLDR is that we trained a multiplayer world model on 10k hours of Rocket League data. We optimized it to be playable at 20fps on a single GPU.
So what you see in the demo is fully generated: there’s no graphics or physics engine. Instead it’s a 5b neural network that takes actions in and gives pixels out.
Having a direct transformation would enable some interesting experiments.
How is the latent state different when everything else stays the same, but you change one physics value, like player one velocity? Is there a cyclical pattern of activation that correlates strongly with the seconds digit of the clock? Can you decode the latent state, give players full boost, and then re-encode it for infinite boost, without losing continuity?
Edit: There sure are a lot of papers on interpretability.
I'd be interested in seeing if fine-tunes that include human gameplay data would be possible.
What is the conversation like within Epic now? Is this still the view? What is the future for simulations like this?
It often feels like the model is ignoring my inputs and just doing what it would expect the bot to do (which is unsurprising if the model could predict what would happen next during training without paying attention to the inputs)