Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
The Transformer replaces recurrent and convolutional layers with a pure attention-based architecture for faster, parallelizable sequence modeling.
How can we replace recurrent and convolutional layers in sequence transduction with a model based entirely on attention mechanisms?
Recurrent neural networks process sequences step-by-step, making them inherently sequential and difficult to parallelize during training. This bottleneck limits performance on long sequences and increases training time significantly. The Transformer discards recurrence entirely, relying instead on multi-head self-attention to relate all positions in a sequence simultaneously. This allows the model to compute representations in parallel across the entire input. On standard machine translation benchmarks, this architecture achieves state-of-the-art results while requiring a fraction of the training time of previous models.
Paper Primer
The Transformer uses a stack of identical layers, each containing a multi-head self-attention mechanism and a position-wise feed-forward network. The core move is the use of Scaled Dot-Product Attention: the model computes a weighted sum of values by comparing queries and keys, scaling the dot products to maintain stable gradients.
Think of the attention mechanism like a library index: the query is the topic you are searching for, the keys are the titles in the catalog, and the values are the actual content of the books. The model calculates how well the query matches each key, then uses those scores to pull the most relevant information from the values.
The Transformer establishes a new state-of-the-art BLEU score for English-to-German translation.
WMT 2014 English-to-German translation task. 28.4 BLEU, outperforming the previous best ensemble models by over 2.0 BLEU.
The architecture significantly reduces training costs compared to recurrent models.
WMT 2014 English-to-French translation task. 41.0 BLEU achieved in 3.5 days on 8 GPUs, at less than 1/4 the training cost of the previous state-of-the-art.
Why move away from recurrent networks if they are the established standard?
Recurrent models factor computation along symbol positions, which prevents parallelization within training examples. The Transformer’s attention mechanism connects all positions with a constant number of sequential operations, enabling massive parallelization.
How does the model handle the order of tokens if it lacks recurrence or convolution?
The model injects positional information by adding sinusoidal "positional encodings" to the input embeddings, allowing it to distinguish between tokens at different positions without needing sequential processing.
Motivation and The Transformer
Why sequence models stall on long inputs and how the Transformer removes that barrier.
Recurrent neural networks (including LSTMs and GRUs) generate a hidden state $h_t$ at each position by conditioning on the previous hidden state $h_{t-1}$ and the current input. This strict left‑to‑right dependency forces the computation to be sequential, which blocks parallel execution across time steps and makes training on long sequences memory‑bound. Even though recent tricks improve efficiency, the fundamental sequential bottleneck remains.
The core obstacle is that each token must wait for the previous token’s hidden state before it can be processed, so the whole sequence becomes a chain of dependent steps.
An RNN processes a sequence token‑by‑token, updating a hidden vector that carries information forward through time.
The shift from recurrence to attention becomes the primary mechanism for sequence modeling.
The Transformer Architecture
The Transformer stacks encoder and decoder layers built from multi‑head attention, feed‑forward nets, and positional encodings.
The model follows the classic encoder‑decoder paradigm: the encoder maps an input sequence $x_1,\dots,x_n$ to continuous representations $z_1,\dots,z_n$, and the decoder generates the output sequence $y_1,\dots,y_m$ one token at a time, conditioning on previously generated tokens.
Compute relevance between queries and keys, scale to keep gradients stable, then blend values according to those relevance scores.
How does the scaling factor differ from plain dot‑product attention?
Plain dot‑product attention omits the $1/\sqrt{d_k}$ term, so when $d_k$ is large the raw scores explode, pushing softmax into near‑one‑hot regimes and yielding vanishing gradients. The scaling term normalizes the variance of the dot products, keeping the softmax in a regime where all keys receive meaningful gradients.
Compute $QK^{\top}=\begin{bmatrix}1&1\\1&-1\end{bmatrix}$.
Scale by $1/\sqrt{2}\approx0.707$: $\begin{bmatrix}0.707&0.707\\0.707&-0.707\end{bmatrix}$.
Apply softmax row‑wise → first row $[0.5,0.5]$, second row $[0.88,0.12]$.
Multiply by $V$: output $=\begin{bmatrix}2.5&2.5\\4.4&0.6\end{bmatrix}$.
The scaling step equalizes the two rows; without it the second row would have produced a near‑one‑hot distribution, ignoring the second key.
Run several independent scaled dot‑product attentions in parallel, each on a different learned subspace, then stitch their results together.
Why not simply increase $d_{\text{model}}$ instead of using multiple heads?
Increasing $d_{\text{model}}$ raises the cost of a single matrix multiplication quadratically, while adding heads keeps each multiplication small ($64\times64$) and exploits parallelism. Moreover, multiple heads let the model attend to different aspects of the data simultaneously, which a single high‑dimensional head cannot express as cleanly.
Project for head 1 with $W_1^Q=W_1^K=W_1^V$ selecting the first two columns → $Q_1=\begin{bmatrix}1&0\\0&1\end{bmatrix}$.
Project for head 2 with $W_2^Q=W_2^K=W_2^V$ selecting the last two columns → $Q_2=\begin{bmatrix}0&1\\1&0\end{bmatrix}$.
Apply scaled dot‑product attention in each head (using the example from the previous block) → outputs $O_1=\begin{bmatrix}2.5&2.5\\4.4&0.6\end{bmatrix}$, $O_2=\begin{bmatrix}2.5&2.5\\0.6&4.4\end{bmatrix}$.
Concatenate $O_1$ and $O_2$ along the feature dimension → $\begin{bmatrix}2.5&2.5&2.5&2.5\\4.4&0.6&0.6&4.4\end{bmatrix}$.
Multiply by $W_O$ (identity for this toy example) → final multi‑head output equals the concatenated matrix.
Even with a tiny model, the two heads produce complementary patterns: head 1 emphasizes the first‑column relationship, head 2 the opposite, illustrating how parallel heads can capture distinct interactions.
Inject a deterministic signal that tells the model each token’s position, because the architecture itself has no recurrence or convolution.
Why not learn the positional embeddings instead of using fixed sinusoids?
Learning adds parameters and may overfit to the training lengths. Fixed sinusoids guarantee a smooth, extrapolatable pattern and performed on par with learned embeddings in the authors’ experiments, so the simpler choice is preferred.
The encoder builds a contextual representation of the source sequence; the decoder consumes that representation while generating the target sequence one token at a time.
What would break if the decoder’s self‑attention were not masked?
Without masking, a token could attend to future tokens that have not been generated yet, leaking information from the future and violating the causal generation constraint. The model would no longer be a proper language model.
**Figure 1.** The Transformer - model architecture.
**Figure 2.** (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several attention layers running in parallel.
Training and Implementation Details
How the model is trained: data handling, hardware, optimizer schedule, and regularization.
The model’s performance hinges on a training pipeline that can keep the massive parallelism of self‑attention while remaining stable and efficient.
We feed the model batches of roughly equal‑length sentences, run many steps on a multi‑GPU machine, and gradually raise the learning rate before letting it decay – a “warm‑up then cool‑down” schedule that lets the network settle before large updates.
Compute the scaling factor $d^{-0.5}=512^{-0.5}\approx 0.044$.
Warm‑up term: $step\_num \, warmup\_steps^{-1.5}=1000 \times 4000^{-1.5}\approx 1000 \times 1.25\times10^{-5}=0.0125$.
Decay term: $step\_num^{-0.5}=1000^{-0.5}=0.0316$.
Take the minimum: $\min(0.0316,0.0125)=0.0125$.
Learning rate at step 1000: $0.044 \times 0.0125 \approx 5.5\times10^{-4}$.
For step 5000, warm‑up term $=5000 \times 4000^{-1.5}\approx 0.0625$, decay term $=5000^{-0.5}=0.0141$; minimum is $0.0141$.
Learning rate at step 5000: $0.044 \times 0.0141 \approx 6.2\times10^{-4}$.
The schedule rises slowly during the first few thousand steps, then the decay term dominates, yielding a modest increase followed by a gentle decline – exactly the shape needed to stabilize deep attention stacks.
Training loop for the Transformer.
Why not use a constant learning rate instead of the warm‑up schedule?
With self‑attention the early layers receive random queries; a large constant step would push those queries into saturated softmax regions, causing gradients to vanish. The warm‑up lets the network first learn a sensible representation before applying larger updates, which empirically yields faster convergence and higher BLEU.
Empirical Performance and Ablations
Transformer sets new translation records while cutting training cost dramatically.
Recurrent and convolutional models process tokens sequentially, limiting speed and long‑range context. The Transformer replaces them with multi‑headed self‑attention, enabling full parallelism and direct global access.
The big Transformer attains 28.4 BLEU on WMT 2014 English‑German, more than 2 BLEU points above the previous best, while using only a fraction of the training compute.
Table 2 shows the big model’s BLEU and its training cost relative to prior architectures.
**Table 2.** The Transformer achieves better BLEU scores than previous state-of-the-art models on the English-to-German and English-to-French newstest2014 tests at a fraction of the training cost.
We next examine how individual design choices affect translation quality (Table 3). Varying the number of attention heads, key/value dimensions, and dropout rates reveals that the default 8‑head configuration with $d_k\!=\!64$ and $d_v\!=\!64$ is near‑optimal.
**Table 3.** Variations on the Transformer architecture. Unlisted values are identical to those of the base model. All metrics are on the English-to-German translation development set, newstest2013. Listed perplexities are per-wordpiece, according to our byte-pair encoding, and should not be compared to per-word perplexities.
Finally, we test the Transformer on English constituency parsing (Table 4). Despite using the same architecture as the translation models, it reaches F1 scores competitive with the best published parsers, especially in the semi‑supervised setting.
**Table 4.** The Transformer generalizes well to English constituency parsing (Results are on Section 23 of WSJ)
Transformer outperforms state‑of‑the‑art models at a fraction of the training cost.
Interpretability and Conclusion
We recap the Transformer’s impact, share visual insights, and note future work.
The Transformer’s fully attention‑based design eliminates sequential bottlenecks, enabling faster training and stronger translation quality, while opening avenues for non‑text modalities and more efficient attention variants.
**Figure 3.** An example of the attention mechanism following long-distance dependencies in the encoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of the verb ‘making’, completing the phrase ‘making...more difficult’. Attentions here shown only for the word ‘making’. Different colors represent different heads. Best viewed in color.
**Figure 4.** Two attention heads, also in layer 5 of 6, apparently involved in an anaphora resolution. Top: Full attentions for head 5. Bottom: Isolated attentions from just the word 'its' for attention heads 5 and 6. Note that the attentions are very sharp for this word.
**Figure 5.** Many of the attention heads exhibit behaviour that seems related to the structure of the sentence. We give two such examples above, from two different heads from the encoder self-attention at layer 5 of 6. The heads clearly learned to perform different tasks.
The full training and evaluation code is released publicly at https://github.com/tensorflow/tensor2tensor, enabling reproducibility and further experimentation.
We thank Nal Kalchbrenner and Stephan Gouws for their insightful comments and suggestions that helped shape this work.