Researchers examined the performance of State Space Models (SSMs) compared to Transformers in sequence modeling. They found that the gating mechanism in SSMs causes them to memorize and delay convergence to in-context learning solutions, affecting their adoption in large-scale language modeling. This study sheds light on the importance of the gating mechanism in SSMs and provides a basis for improving linear-time models.