attention mechanism

In the context of machine learning, attention (or self-attention) is a technique that mimics the effect of the cognitive process of attention in organisms. They are used in a wide variety of machine learning models, including in natural language processing and computer vision.

Transformer networks make extensive use of attention mechanisms to achieve their expressive power. Computer vision systems based on convolutional neural networks can also benefit from attention mechanisms.^{[citation needed]}

The two commonest attention techniqes used are dot-product attention, which uses the dot product between vectors to determine attention, and multi-head attention, which combines several different attention mechanisms to direct the overall attention of a network or sub-network.