AI & LLMs · Frontier
注意力机制(Attention)
模型读一句话时,会自动算出哪些词跟哪些词关系更紧。它是主流大模型的核心部件,也是模型能抓住长句上下文的原因。
Some fields on this page have no English version yet — they are shown in the original Chinese.
TL;DRIn 30 seconds
- 模型读一句话时,会算出哪些词跟哪些词关系更近,这个打分的部件就是它。
- 有了它,句子再长,前后的词也能直接扯上关系,不用一个字一个字传。
- 名字像人的专注,其实只是一组数值,没有任何主观成分。
OverviewIn detail
注意力机制是当前大语言模型的核心构件。模型在理解某个词的时候,会回头看整句里其他所有的词,算出每一个该分到多少关注。读“小明把书借给了小红,因为她在复习”时,判断“她”究竟指谁,靠的就是这种关联打分。分数高的词,对当前这个词的理解影响就更大。
在它之前,模型按顺序一个字一个字读,读到后半句往往已经忘了前半句,句子一长就乱套。注意力机制让任意两个位置之间可以直接建立联系,不必隔着整串字符慢慢传递,处理长文本的能力因此上了一个台阶,这也是现在的模型能处理整本书那么长输入的原因。
它的名字来自一篇论文,标题直译过来是“注意力就是你所需要的一切”,此后几乎所有主流模型都建在这套结构之上。很多后续改进都围绕着让它算得更快、更省内存。普通用户不需要直接操作它,但模型能读懂长段落、能接住前文的指代,根子都在这里。
需要澄清的是,这里的注意力没有任何主观成分。它只是给不同词算出的一组数值,表示相关的程度高低,模型并不真的在专注,也不在思考。把它理解成数学上的权重分配,比理解成人的注意力更接近实情。理解到这一层,对日常使用已经足够了。
MythsCommon misunderstandings
- ✕以为模型真在专注某处 ✓它算的只是相关度数值,没有意识,也没有在看什么
- ✕以为有了它前文就不会忘 ✓关联算得再好也有漏掉的地方,句子太长仍可能抓不住重点
- ✕以为这只是个可有可无的小技巧 ✓几乎所有主流模型都建在它上面,长文本能力根子在这里
In practiceWhere you will meet it
- 让它读一份长报告并回答里面的细节,它能接住前后文的指代
- 发现它把代词认错了人,就明白关联也会算错
- 看到模型介绍强调能处理超长文本,知道背后靠的是这套机制
SummaryThe key points
- 让模型判断句子里哪些词关系更紧密
- 是几乎所有主流大语言模型的基础结构
- 让模型能处理长文本、接住前文的指代
- 它是算出来的数值权重,不含意识
FAQ
What does 注意力机制(Attention) mean?
模型读一句话时,会算出哪些词跟哪些词关系更近,这个打分的部件就是它。有了它,句子再长,前后的词也能直接扯上关系,不用一个字一个字传。名字像人的专注,其实只是一组数值,没有任何主观成分。
Where will I run into 注意力机制(Attention)?
让它读一份长报告并回答里面的细节,它能接住前后文的指代;发现它把代词认错了人,就明白关联也会算错;看到模型介绍强调能处理超长文本,知道背后靠的是这套机制。
What do people most often get wrong about 注意力机制(Attention)?
以为模型真在专注某处 — but 它算的只是相关度数值,没有意识,也没有在看什么;以为有了它前文就不会忘 — but 关联算得再好也有漏掉的地方,句子太长仍可能抓不住重点。
What are the key points about 注意力机制(Attention)?
让模型判断句子里哪些词关系更紧密;是几乎所有主流大语言模型的基础结构;让模型能处理长文本、接住前文的指代;它是算出来的数值权重,不含意识。
Common pitfalls 这里的注意力只是数学上的相关度分数,跟人的专注完全是两回事。
Also written asYou may also see it written as
These are all the same concept — the same thing, however it is written in a skill description. Do not mistake different spellings for different ideas.
Entry 26 of 186 · All entries
Keep readingRelated reading
AI Agent
An AI program that sets its own goal, breaks it into steps and calls tools to get the job done. You hand it a task and it works out how to do it — instead of only answering one question at a time.
LLM
A very large program that learned to predict the next piece of text after reading an ocean of it. Whatever you type, it is guessing what is most likely to come next — writing, translating and answering questions are all the same thing used differently.
嵌入(Embedding)
把一段文字换算成一串数字,让电脑能比较哪两句话意思更接近。搜索和知识库能在换了说法时也找对答案,靠的就是它。
Claude
A large language model built by Anthropic, known for understanding long documents, writing well and writing code. It comes both as a chat assistant on the web and phone, and in a dedicated form for programming.
Prompt
The message you send to an AI — the instruction or question you give it. How well that passage is written decides whether the answer that comes back is on target.
MCP
A standard interface that lets AI connect to outside tools and data in one uniform way. Build a tool to it once and any AI assistant can call it, saving you from writing a separate integration for each tool.
Where to go nextExplore more
Back to the skills themselves
Most of the concepts covered here come from skills that install into your AI client with a single command.