Self Attention自注意力机制

李宏毅: 《机器学习》 学习笔记

Vector Set as Input

一段文本

一段声音信号

图

社交图

分子图

---

---

Output

each vector has a label(sequence labeling)

the whole sequence has a label

model decides the num of labels itself(seq2seq)

---

---

self-attention

以sequence labeling 为例:

self-attention

Truncated self-attention

通常我们不需要self-attention 一整个句子,只关注其中一部分，这样可以加快运算的速度。

self-attention for Image

self-attention GAN

DEtection Transformer(DETR)