Paperlane

no. 17 · Wednesday 18 November 2026

Attention's forward pass and Frank-Wolfe

Albert Alcalde, Borjan Geshkovski and Domènec Ruiz-Balet

math.OC · v3 · code

Abstract

We study the hardmax limit of self-attention dynamics for token embeddings obtained in the zero-temperature ($β\to+\infty$) regime, and relate it to the finite-$β$ setting. In this limit, the update rule can be viewed as a Frank-Wolfe step for a quadratic objective over the convex hull of the current token embeddings. When the key-query matrix is negative semidefinite, the method linearly contracts all tokens to a single cluster at the origin. When it is positive semidefinite, extending the hardmax rule to the entire convex hull induces a Voronoi diagram: vertices are stationary, interior points remain in their initial cells, and each token moves along a straight line toward its cell's vertex, yielding (super-)exponential convergence. As a byproduct, we also establish well-posedness of the associated ODE limit in this regime. Returning to the finite-$β$ regime, we model self-attention dynamics as a Markov chain and prove dynamic metastability: with high probability, interior tokens reach near-vertex configurations in a constant number of steps and remain within a small neighborhood for times that grow exponentially in the inverse temperature $β$, before ultimately collapsing to the origin. Thus, the hardmax dynamics accurately approximate the finite-$β$ process over exponentially long time horizons.

code
github.com/borjanG/2025-transformers-frank-wolfe
comments
Accepted at the 43rd International Conference on Machine Learning: https://icml.cc/virtual/2026/poster/60480 Funded by the European Union's HORIZON-MSCA-2021-DN-01-01, through ModConFlex ("Modelling and control of flexible structures interacting with fluids"; No. 101073558; DOI: 10.3030/101073558). Code: https://github.com/borjanG/2025-transformers-frank-wolfe, code DOI: 10.5281/zenodo.23468440
fields
Optimization and Control
pdf ↗ page arXiv:2508.09628 ↗