How Attention Works in Transformers: Queries, Keys and Values Explained

Attention is the mechanism that lets a transformer decide which parts of its input matter for each position. When the model processes a word, it can look at every other word in the sequence and pull in information from the ones that are most relevant. This article explains the query, key and value computation, the softmax weighting that follows, and how multiple attention heads and causal masks fit around it.

The core problem attention solves

Earlier sequence models, such as recurrent neural networks, read text one token at a time and passed a running summary forward. Information from early words had to survive many steps to influence later ones, and long-range dependencies were hard to learn. Attention removes that bottleneck. Every token can access every other token directly, so the distance between related words matters less.

Consider the sentence "The animal did not cross the street because it was too tired." To interpret "it", the model needs to connect the pronoun back to "animal", which may be many tokens away. Attention provides a direct route for that connection.

Queries, keys and values

For each token, the model computes three vectors by multiplying its current representation by three learned weight matrices. These are the query, the key and the value.

• The query describes what the token is looking for.
• The key describes what the token offers to be found by others.
• The value holds the information the token will contribute if it is selected.

To decide how much token i should attend to token j, the model takes the dot product of i's query with j's key. A large dot product means the key matches the query well. The dot products for one token against all others form a row of scores.

Scaled dot-product attention

The scores are divided by the square root of the key dimension, then passed through a softmax function. Softmax converts scores into positive weights that sum to one. The output for token i is the weighted sum of all value vectors, using those weights.

The full formula from the 2017 paper "Attention Is All You Need" is softmax(Q K transpose divided by the square root of d_k) times V. Dividing by the square root of the key dimension keeps the dot products from growing too large, which would push softmax into regions with tiny gradients.

A small numerical example

Suppose one token produces raw scores of 2, 1 and 0 against three tokens. Softmax exponentiates each score and normalizes them. The exponentials are about 7.389, 2.718 and 1.000, which sum to about 11.107. The resulting weights are about 0.665, 0.245 and 0.090.

Token	Raw score	Softmax weight	Interpretation
Token A	2.0	0.665	Strongest match, dominates the output
Token B	1.0	0.245	Moderate contribution
Token C	0.0	0.090	Small contribution

The output vector is 0.665 times the value of A, plus 0.245 times the value of B, plus 0.090 times the value of C. The representation of the current token is therefore a blend of the most relevant context, and the blend changes with every input.

Multi-head attention

A single attention pattern can capture only one kind of relationship at a time. Multi-head attention runs several attention operations in parallel, each with its own query, key and value matrices. One head might track grammatical agreement, another might track which word a pronoun refers to, and another might focus on nearby words.

The outputs of the heads are concatenated and projected back to the model dimension. Splitting the representation into heads does not multiply the total computation by the number of heads, because each head works in a smaller dimension. The overall cost stays comparable to a single large attention operation.

Causal masks for generation

A language model that generates text must not peek at future tokens during training. Causal masking handles this by setting the scores for all future positions to negative infinity before the softmax. Those positions then receive zero weight. Each token can attend only to itself and earlier tokens, which matches how text is produced left to right.

Masking also allows a transformer to be trained on every position of a sequence at once. Without it, the model would need one training step per position, which would be far slower.

Positional information

Attention by itself does not know word order. Without position information, attention treats the input as a set: shuffling the tokens only shuffles the outputs, so the model needs an explicit signal for position. Early transformers added sinusoidal positional encodings to the input embeddings. Later models use learned position embeddings or rotary position embeddings, which encode relative offsets inside the attention computation. The choice affects how well a model handles sequences longer than it saw in training.

The cost of attention

Every token compares itself with every other token, so the number of score computations grows with the square of the sequence length. Doubling the context roughly quadruples the attention cost. This is one reason long context windows are expensive, and it has motivated many efficiency methods, from sparse attention patterns to caching of keys and values during generation.

Property	Recurrent network	Transformer self-attention
Processing order	Sequential, one step at a time	Parallel across positions in training
Path length between distant tokens	Grows with distance	One attention step
Main cost driver	Number of steps	Sequence length squared
Long-range dependencies	Hard to preserve	Direct access

Residual connections and normalization

Attention is only one part of a transformer block. The attention output is added back to the block's input through a residual connection, so each layer adds information rather than replacing the whole representation. This keeps gradients flowing through deep stacks and lets the model default to passing information along unchanged when a layer has nothing useful to add. Layer normalization rescales each token's vector to a stable range before the attention and feed-forward sublayers. Together, residual connections and normalization make it practical to stack dozens of blocks.

Each block follows attention with a feed-forward network applied to every position independently. Attention moves information between tokens, while the feed-forward network transforms each token's representation. Researchers often discuss these two roles separately, although the division is a useful simplification rather than a strict rule.

Key takeaways

Attention lets each token build its representation from the other tokens it needs. Queries and keys determine the matching weights, and values supply the content that gets mixed together.

Softmax turns scores into weights that sum to one, so each output is a weighted average of value vectors. Multi-head attention runs several such patterns in parallel to capture different relationships.

Causal masks keep generation honest by blocking access to future tokens. The quadratic cost in sequence length is the main price of this design, and it shapes how modern models handle long inputs.