Knowledge Distillation

Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.

synthetic-sciences Updated 4.4k repo stars
An OpenScience session comparing EGFR inhibitor potency: the trace shows the search, the written files, the analysis run and the answer with a results table

File contents

synthetic-sciences/openscience commit 0fb72ed71b

Frequently asked questions

npx get-skill add synthetic-sciences/knowledge-distillation