Beyond Attention: Grassmann Layers Rival Transformers
Researchers challenge the assumption that self-attention is essential for sequence modeling performance. They introduce a competing architecture based on Grassmann manifolds that performs comparably to standard transformers while offering clearer mathematical interpretation.
The proposed Causal Grassmann layer replaces the typical attention matrix with a geometric approach using Plucker coordinates. Instead of computing pairwise interactions in an unstructured tensor space, the system encodes local token relationships as subspaces on a Grassmann manifold, then fuses these features back into hidden states through gated mixing. This keeps computation on a mathematically structured manifold rather than in an opaque tensor space, potentially enabling better understanding of model behavior.
On language modeling benchmarks, attention-free Grassmann models with 13-18M parameters achieve perplexities within 10-15% of transformer baselines. On the SNLI inference task, a Grassmann-based variant actually outperforms the transformer baseline. Beyond performance parity, the authors argue this manifold-based approach creates a more structured path toward geometric interpretability and invariant-based reasoning in neural networks.
Read the full article
Continue reading at Hacker News →This is an AI-generated summary. Read the original for the full story.