What I Learned Implementing On-Policy Distillation
An experiment distilling Qwen3-8B into a 0.6B student: the mathematics of teacher feedback, the improvement greedy evaluation nearly missed, and what a failed training run revealed.
An experiment distilling Qwen3-8B into a 0.6B student: the mathematics of teacher feedback, the improvement greedy evaluation nearly missed, and what a failed training run revealed.