Teacher outputs, probabilities, intermediate states, and labeled data can supply the student's targets. The student may be faster or smaller, but the retained behavior depends on its capacity and the coverage of distillation data.
Model distillation creates a deployment model by training it to imitate selected behavior of a larger or more capable model.
Teacher outputs, probabilities, intermediate states, and labeled data can supply the student's targets. The student may be faster or smaller, but the retained behavior depends on its capacity and the coverage of distillation data.