Research paper discussion
26
THU
Thu, Nov 26
04:15 PM - 06:00 PM
No tickets required
Seminar and Discussion on Model Distillation in the Era of Large Language Models
Distillation has made headlines as a way for one AI model to learn from another, with or without the consent of the model provider. Beyond the headlines, it is a fundamental concept that every ML engineer or researcher should be familiar with. It is an essential part of the pursuit of efficient neural networks and can therefore be found in many places where models are deployed to production.
The event is open to both relative beginners and those who want to explore distillation in more depth. In the first half, we'll cover Hinton et al.'s classic paper on knowledge distillation and look at what a smaller model can learn from a teacher’s predictions. In the second half, we'll discuss the more advanced concept of generalized knowledge distillation and the paper On-Policy Distillation of Language Models, also coming out of Google.
Please check out the papers beforehand. Both of them are a good and concise read. However, you can also check out these articles if you do not find them approachable:
https://www.ibm.com/think/topics/knowledge-distillation (classic distillation)
https://thinkingmachines.ai/blog/on-policy-distillation/ (distillation of language models)
Location: Ångström laboratory, Room 101162
Date: 26 November
Time: 17.15-19.00
sign up here: https://luma.com/dt9kt6mq