The Recurrent Looped Transformer (RLT) represents a significant departure from standard decoder-only LLM architectures. The core innovation lies in maintaining the decoder’s complete state – including the sliding-window attention (SWA) cache – across every token in both the prompt and the response. This eliminates the typical boundary reset found in conventional models, enabling a continuous temporal chain. The proposed design utilizes a causal encoder paired with a recurrent decoder, processing tokens in parallel and maintaining a shared key-value memory.
In a reference configuration, the RLT employs 48 tied encoder and decoder layers, resulting in 96 logical blocks per token. This parameter reuse, rather than activation copying, is central to the RLT’s operation. The model’s state path grows linearly with the sequence length, offering the potential for unbounded temporal depth. However, the report cautions that gating mechanisms and contraction could limit this depth, and structural depth alone doesn’t guarantee improved reasoning.
Beyond the architecture, the RLT incorporates several key design choices. The model-hardware co-design leverages token-parallel kernels for encoder features and memory projections, while maintaining sequential transitions within the decoder. Batching, kernel fusion, and checkpointing are identified as potential implementation targets. The RL replay mechanism further enhances the system, recording behavior log-probabilities under the sampling distribution, and rebuilding the state from the start with each replay.
Crucially, the RLT’s design addresses multi-turn serving scenarios. A fixed-weight snapshot allows for reuse, as the state is independent of the serving split. External tokens in multi-turn RL updates the state but doesn’t affect importance-ratio factors. The RLT builds upon existing research, drawing inspiration from Feedback Transformer, Recurrent Transformer, YOCO, and DeepSeek-V4.1-Flash. The research is currently without measured results, focusing on validation targets such as reasoning quality and scaling.
