This is one of those papers that ought to have several textbook chapters unpacking some of the insights... I'll just pick a couple things I love:
1) There's a kind of "rabbit–duck illusion" moment where they show how you can reframe all the linear algebra around attention in a totally different way, demoting the Q, K, and V matrices in favor of emphasizing a set of much larger matrices that are mathematically equivalent and very useful for interpretability purposes. (So, don't trust anyone who insists that vague analogies about "keys" and "queries" are the only good way to understand attention! They probably haven't read this paper.)
2) They show the importance of the "residual stream" inside LLMs: it's not just a series of bypasses of layers that's useful for training stability (like I first had it explained to me) but a kind of main communication bus that runs unbroken through the whole model from embedding to output. Attention heads and Feedforward are off to the side, adding embeddings into the stream. Somehow I find this a much more satisfying way to understand LLM architectures, and whenever I see an architecture diagram now I mentally redraw it with the residual stream at the center.
myself248 2 days ago [-]
Nothing about B-H curves or magnetization current, must be that other kind of transformer, in that other kind of circuit.
greenbit 2 days ago [-]
Someone should step up and rectify that, there's a lot of potential for confusion.
ziofill 2 days ago [-]
This comment is hysteresical
Bluestein 2 days ago [-]
I find it polarizing.-
thatspartan 2 days ago [-]
It can be quite eddyfying
pooyamo 2 days ago [-]
A harmonious comment about an impedance mismatch quickly spiraled into a galvanizing flow
greenbit 1 days ago [-]
It's just that HN is currently saturated with this kind of thing
MisterTea 1 days ago [-]
Regardless of flux, we still haven't addressed the core issue.
_jayhack_ 2 days ago [-]
Shocking to me how little interest the general public has in mechinterp given the alien capabilities demonstrated by LLMs - this and the subsequent transformer-circuits.pub publications will be seen as classic, foundational work in a few years
amelius 2 days ago [-]
> A previous project, the Distill Circuits thread, has attempted to reverse engineer vision models, but so far there hasn’t been a comparable project for transformers or language models.
1) There's a kind of "rabbit–duck illusion" moment where they show how you can reframe all the linear algebra around attention in a totally different way, demoting the Q, K, and V matrices in favor of emphasizing a set of much larger matrices that are mathematically equivalent and very useful for interpretability purposes. (So, don't trust anyone who insists that vague analogies about "keys" and "queries" are the only good way to understand attention! They probably haven't read this paper.)
2) They show the importance of the "residual stream" inside LLMs: it's not just a series of bypasses of layers that's useful for training stability (like I first had it explained to me) but a kind of main communication bus that runs unbroken through the whole model from embedding to output. Attention heads and Feedforward are off to the side, adding embeddings into the stream. Somehow I find this a much more satisfying way to understand LLM architectures, and whenever I see an architecture diagram now I mentally redraw it with the residual stream at the center.
How successful was that?
- https://distill.pub/2021/distill-hiatus/
... and, here we are.-
https://youtu.be/KV5gbOmHbjU?is=GSiv0rSHBocNdU8l
Also long, but it's detailed and complicated, so there is no escaping that.
https://en.wikipedia.org/wiki/Royal_Road#A_metaphorical_%22R...