Effective Teacher Support for Reasoning Distillation
arXiv:2611.14850v1 Announce Type: new Abstract: Forward-KL language-model distillation normally computes dense teacher and student scores over the full vocabulary at every supervised position, making training and stored teacher targets expensive. We instead distill from the teacher's highest-probability vocabulary fraction, renormalizing the retained mass as we sweep support from 100% to 0.01%. Our main study distills Qwen3 4B -> 1.7B. We confirm the broader support-reduction finding on larger Qwen3 8B -> 4B and Nemotron Nano 12B -> 9B pairs and under answer-only supervision. We evaluate gener
阅读原文 · arXiv 人工智能本文由 SoccerDream AI 自动聚合索引,内容版权归原作者与 arXiv 人工智能 所有。
本站仅作技术动态聚合,不构成任何形式的建议。
本站仅作技术动态聚合,不构成任何形式的建议。