Generative Denoise Distillation: Simple stochastic noises induce efficient knowledge transfer for dense prediction

Knowledge distillation is the process of transferring knowledge from a more powerful large model (teacher) to a simpler counterpart (student). Numerous current approaches involve the student imitating the knowledge of the teacher directly, which tend to learn each spatial location’s features indiscriminately. Real-world datasets frequently exhibit noise, motivating models to acquire compact and representative features instead of memorizing …