Sebastian Raschka examines the architectural foundations of modern large language models, focusing on DeepSeek V3 and its successor R1. Published in July 2025, the article contrasts structural changes from GPT-2 (2019) to models like Llama 4 and DeepSeek V3 (late 2024). The text notes that while positional embeddings shifted to RoPE and activations moved to SwiGLU, core architectural similarities remain.
Grouped-Query Attention vs Multi-Head Attention
The article reviews Grouped-Query Attention (GQA) as a standard replacement for Multi-Head Attention (MHA). Unlike MHA where each head has unique keys and values, GQA groups heads to share key-value projections. This design reduces the number of computations required for key and value tensors during inference.
Multi-Head Latent Attention in DeepSeek V3
DeepSeek V3 utilizes Multi-Head Latent Attention (MLA) instead of GQA. MLA compresses key and value tensors into a lower-dimensional space before storing them in the KV cache. At inference time, these compressed tensors are projected back to their original size for use.
Performance and Memory Trade-offs
Ablation studies from the DeepSeek-V2 paper suggest MLA offers better modeling performance than MHA, whereas GQA appears to perform worse. This likely explains the choice of MLA in DeepSeek V3. The technique reduces KV cache memory usage while adding an extra matrix multiplication during inference.
Source: https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison
