Mamba represents an alternative architecture to Transformers using State Space Models (SSMs). It aims to match Transformer performance while supporting feasible sequence lengths of approximately one million tokens by removing the quadratic complexity bottleneck inherent in Attention mechanisms.
The core problem with Transformers is that every token can attend to all previous tokens, creating a forward pass time complexity of O(n²) and requiring O(n) space for KV cache storage. This leads to significant latency increases and memory constraints as context size grows.
Mamba addresses this by substituting the Attention component with an SSM while retaining MLP-style projections for computation. The architecture uses differential equations to model state evolution, allowing the system to infer future states from current observations without processing the entire history.
On language modelling tasks using The Pile dataset, a Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size in both pretraining and downstream evaluation. This architecture achieves state-of-the-art performance across modalities including language, audio, and genomics.
