AI & LLMs · Frontier
蒸馏(Distillation)
一种训练方法:让一个小模型大量模仿大模型的回答,把本事浓缩过来。目的是在尽量保住效果的前提下,把模型做小做快、少花钱。
Some fields on this page have no English version yet — they are shown in the original Chinese.
TL;DRIn 30 seconds
- 一种训练方法:让一个小模型大量模仿大模型的回答,把本事浓缩过去。
- 目的是把模型做小做快又少花钱,好塞进手机或被高频调用。
- 学生通常接近老师但略逊一筹,尤其在没见过的题上。
OverviewIn detail
知识蒸馏是一种训练方法。先有一个又大又强、但跑起来很贵的模型,称为老师;再让一个小模型当学生,去大量模仿老师的回答。学生学到的不是标准答案,而是老师解题的思路,所以常常比自己闷头训练要强。
大模型准确但笨重,放到手机里,或者要同时服务很多人时就太贵。蒸馏想在尽量保住本事的前提下,把模型做小、做快,省内存也省电,让它能在更多地方跑得动。
你看到某个工具说“这个模型是从大模型蒸馏来的”,意思是它是个小个子版本,能力接近老师但更轻便,适合本地运行或者被高频调用。手机、浏览器插件,或者要同时接待很多用户的场景,通常就靠这类小模型撑着。
蒸馏不是“把模型压缩一下”那么简单,它需要重新训练一个学生模型。学生也不可能完全达到老师的水平,通常是接近但略逊。它和量化不是一回事:量化是压缩已经训好的模型,蒸馏是重新训出一个,目的都是让模型更轻,走的却是两条完全不同的路。
常见的做法不只是让学生照着正确答案学,还让它对齐老师的“偏好”,也就是老师更倾向于选哪个词。这样学出来,学生连老师的分寸感都能沾上一点,这也是它比单纯拿标准答案训练更有效的地方。
MythsCommon misunderstandings
- ✕以为蒸馏就是把大模型压扁一点 ✓压缩是给同一个模型瘦身,蒸馏是重新训练出另一个小模型
- ✕以为学生能超过老师 ✓它是照着老师的回答学的,上限就在老师那一档,通常还低一点
In practiceWhere you will meet it
- 看到某个模型标着“由大模型蒸馏而来”,就知道它是轻便版
- 要在手机上做不联网的问答时,选这类小模型
- 同一件事要被反复调用几万次时,用它压住成本
SummaryThe key points
- 让小模型大量模仿大模型的回答
- 目的是把模型做小做快、省成本
- 学生通常接近老师但略逊一筹
- 和压缩已有模型的量化是两回事
FAQ
What does 蒸馏(Distillation) mean?
一种训练方法:让一个小模型大量模仿大模型的回答,把本事浓缩过去。目的是把模型做小做快又少花钱,好塞进手机或被高频调用。学生通常接近老师但略逊一筹,尤其在没见过的题上。
Where will I run into 蒸馏(Distillation)?
看到某个模型标着“由大模型蒸馏而来”,就知道它是轻便版;要在手机上做不联网的问答时,选这类小模型;同一件事要被反复调用几万次时,用它压住成本。
What do people most often get wrong about 蒸馏(Distillation)?
以为蒸馏就是把大模型压扁一点 — but 压缩是给同一个模型瘦身,蒸馏是重新训练出另一个小模型;以为学生能超过老师 — but 它是照着老师的回答学的,上限就在老师那一档,通常还低一点。
What are the key points about 蒸馏(Distillation)?
让小模型大量模仿大模型的回答;目的是把模型做小做快、省成本;学生通常接近老师但略逊一筹;和压缩已有模型的量化是两回事。
Common pitfalls 它是重新训练一个小模型,不是把大模型简单压缩一下,两者别弄混。
Entry 13 of 186 · All entries
Keep readingRelated reading
AI Agent
An AI program that sets its own goal, breaks it into steps and calls tools to get the job done. You hand it a task and it works out how to do it — instead of only answering one question at a time.
LLM
A very large program that learned to predict the next piece of text after reading an ocean of it. Whatever you type, it is guessing what is most likely to come next — writing, translating and answering questions are all the same thing used differently.
嵌入(Embedding)
把一段文字换算成一串数字,让电脑能比较哪两句话意思更接近。搜索和知识库能在换了说法时也找对答案,靠的就是它。
Claude
A large language model built by Anthropic, known for understanding long documents, writing well and writing code. It comes both as a chat assistant on the web and phone, and in a dedicated form for programming.
Prompt
The message you send to an AI — the instruction or question you give it. How well that passage is written decides whether the answer that comes back is on target.
MCP
A standard interface that lets AI connect to outside tools and data in one uniform way. Build a tool to it once and any AI assistant can call it, saving you from writing a separate integration for each tool.
Where to go nextExplore more
Back to the skills themselves
Most of the concepts covered here come from skills that install into your AI client with a single command.