Authors: Zayan Hasan, Aniketh Malipeddi, Aneesh Chatrathi
State Space Models (SSMs) have emerged as a new linear computational complexity transformer rival to sequence modeling on long sequences with competitive performance. The dynamics of training and stability properties of SSMs remain poorly understood from a spectral perspective. This work presents the first wide reaching spectral analysis of SSM based language models. Providing a systematic framework to examine how eigenvalue distributions and spectral radii evolve during training, through experiments on a 737K parameter SSM model having 3 layers, state space dimension 128, and model space dimension 8, it was discovered that although the minority of the state matrices learned lead to theoretical spectral stability with mean spectral radius 1.078. The model demonstrates excellent convergence however, reducing training loss from 3.127 to 0.305 using 100 epochs. The eigenvalue analysis demonstrates common clustering in the negative real axis with concentration centered about negative 0.8, exhibiting a bimodal spectral radius distribution exhibiting systematic behavior in SSM dynamics. The key result portrays that SSMs operate efficiently in scenarios such as these. The selective mechanism provides adaptive control that prevents mathematical instabilities from causing training divergence. This renders assumptions of classical neural network stability hard to maintain and makes spectral analysis an essential for understanding similar model behavior. This work provides practical insight toward constructing more principled, stability aware designs for such models and frameworks.
Comments: 5 Pages. (Note by viXra Admin: Please submit article written with AI assistance to ai.viXra.org)
Download: PDF
[v1] 2025-09-19 18:24:16
Unique-IP document downloads: 258 times
Vixra.org is a pre-print repository rather than a journal. Articles hosted may not yet have been verified by peer-review and should be treated as preliminary. In particular, anything that appears to include financial or legal advice or proposed medical treatments should be treated with due caution. Vixra.org will not be responsible for any consequences of actions that result from any form of use of any documents on this website.
Add your own feedback and questions here:
You are equally welcome to be positive or negative about any paper but please be polite. If you are being critical you must mention at least one specific error, otherwise your comment will be deleted as unhelpful.