The student learns from teacher probabilities, generated examples, intermediate features, ground-truth labels, or a combination. Distillation can reduce serving cost while preserving selected capabilities, but it may also transfer teacher errors.
Knowledge distillation trains a smaller or simpler student model to reproduce behavior learned by a teacher model.
The student learns from teacher probabilities, generated examples, intermediate features, ground-truth labels, or a combination. Distillation can reduce serving cost while preserving selected capabilities, but it may also transfer teacher errors.