The research introduces a method for pruning transformer models by framing it as a damage-aware multi-armed bandit problem. Attention heads and MLP channel groups are temporarily masked during calibration, measuring the resulting loss degradation. This paired damage metric is used to guide the selection process, employing either a UCB-style policy or fractional-Beta Thompson Sampling. The final mask is constructed sequentially, adding units based on the reward signal. The selected units are functionally zeroed in the original checkpoint, representing structural suppression rather than physical compression.
Experiments were conducted on WikiText-2, LAMBADA, and Imagenette using models including GPT-2, OPT, Pythia, Qwen2.5, SmolLM2, ViT-B/16, DeiT-Tiny, and Swin-Tiny. Comparisons were made against random, magnitude, static-saliency, and budgeted-greedy selection. Across five seeds, the bandit methods generally reduced degradation relative to budgeted greedy in language model comparisons.
Statistical analysis of 28 comparisons revealed significant gains. 23 bootstrap confidence intervals excluded zero, and 11 paired tests had p < 0.05. Matched-evaluation results for ViT-B/16 and Swin-Tiny indicated that the gains were not solely due to a larger candidate-evaluation budget.
This approach offers a structured method for pruning transformers, potentially reducing model size while maintaining performance. The use of a bandit algorithm allows for adaptive selection based on observed damage, leading to more effective pruning than traditional methods.
Source: https://arxiv.org/abs/2609.05448