RAPID introduces a new approach to inter-example relational distillation. The method separates a reliability-gated target from a full support adaptive pair proposal. This allows for selective emphasis on teacher relations based on a reliability metric. Calibrated teacher entropy and detached student-teacher residuals are used to evaluate the selected relations. The system employs exact inverse proposal correction to ensure conditional unbiased loss and gradient estimators within the gated mini batch target.
Experiments were conducted in two text classification settings: AG News with BERT-to-DistilBERT distillation and SST-2 with DistilBERT-to-DistilBERT distillation. Each setting utilized three paired seeds and a relation budget of either 256 or 64. The results demonstrated that the gated target yielded the highest mean accuracy. Specifically, on AG News, RAPID achieved 94.285 plus or minus 0.054 percent mean accuracy, while on SST-2, it reached 88.800 plus or minus 0.532 percent.
Compared to a cross-entropy baseline, RAPID ranked second in both datasets. The baseline achieved 94.154 plus or minus 0.124 percent on AG News and 87.271 plus or minus 0.162 percent on SST-2. These results support a modular design where target reliability and evaluation priority are separable design dimensions.
Source: https://arxiv.org/abs/2609.05481