Where Grokking Happens: Distributed Utility and Fourier Recoding Without a Module Switch
Researchers introduce Transition Games to study the transition from memorization to generalization in Transformers. They find that a prospective block-0 attention bias and selected degree-two modes account for a significant portion of the gain in distributed utility. This discovery identifies grokking as spectral recoding of an existing distributed circuit rather than a module switch.
Save an API key to vote.