Paperlane

no. 17 · Wednesday 18 November 2026

Clustering in pure-attention hardmax transformers and its role in sentiment analysis

Albert Alcalde, Giovanni Fantuzzi and Enrique Zuazua

math.DS · v3 · code

Abstract

Transformers are extremely successful machine learning models whose mathematical properties remain poorly understood. Here, we rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as the number of layers tends to infinity. By viewing such transformers as discrete-time dynamical systems describing the evolution of points in a Euclidean space, and thanks to a geometric interpretation of the self-attention mechanism based on hyperplane separation, we show that the transformer inputs asymptotically converge to a clustered equilibrium determined by special points called \textit{leaders}. We then leverage this theoretical understanding to solve sentiment analysis problems from language processing using a fully interpretable transformer model, which effectively captures `context' by clustering meaningless words around leader words carrying the most meaning. Finally, we outline remaining challenges to bridge the gap between the mathematical analysis of transformers and their real-life implementation.

code
github.com/DCN-FAU-AvH/clustering-hardmax-transformers
comments
Funded by the European Union's HORIZON-MSCA-2021-DN-01-01, through the ModConFlex project ("Modelling and control of flexible structures interacting with fluids''; Grant Agreement No. 101073558; DOI: https://doi.org/10.3030/101073558). Accompanying code available at: https://github.com/DCN-FAU-AvH/clustering-hardmax-transformers, code DOI: https://doi.org/10.5281/zenodo.21107513
journal
SIAM Journal on Mathematics of Data Science 7(3):1367-1393, 2025
fields
Dynamical Systems
pdf ↗ page arXiv:2407.01602 ↗