Knowledge Distillation

Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.

bishal-pdMSFT Updated 0 repo stars

File contents

bishal-pdMSFT/msb-skills commit c79e89c6cf

Frequently asked questions

npx get-skill add bishal-pdmsft/knowledge-distillation