Encoder-Decoder

The encoder-decoder (seq2seq) architecture maps a variable-length input sequence to a variable-length output sequence by first compressing the input into a representation and then generating the output one token at a time conditioned on that representation.

The Architecture

An encoder reads the entire source sequence and produces a set of contextual representations. A decoder generates the target sequence autoregressively, attending to both its own previous outputs and the encoder representations.

In the earliest RNN seq2seq models the encoder squeezed the whole input into a single fixed context vector (its final hidden state), which the decoder consumed. This bottleneck hurt long inputs and motivated attention, where the decoder can look back at every encoder state instead of a single summary.

Context vector vs attention

The fixed context vector forces all information through one vector of size $d$. Cross-attention removes that bottleneck: the decoder computes a fresh weighted read over all encoder states at every step, so information capacity scales with source length.

Cross-Attention

In the Transformer encoder-decoder, the decoder has two attention sublayers per block:

1. Masked self-attention over the target tokens generated so far (causal).
2. Cross-attention, where the queries come from the decoder while the keys and values come from the encoder output:

$\text{CrossAttn}(H_{\text{dec}},H_{\text{enc}})=\text{softmax}\left(\frac{(H_{\text{dec}}{W}^Q)(H_{\text{enc}}{W}^K{)}^{\top }}{\sqrt{d_k}}\right)(H_{\text{enc}}{W}^V)$

This is exactly Self-Attention with a split source of queries versus keys/values.

Teacher Forcing and Autoregressive Decoding

During training we use teacher forcing: the decoder is fed the ground-truth previous tokens rather than its own predictions, which stabilizes learning and lets the whole target sequence be processed in parallel with a causal mask. At inference we instead decode autoregressively, feeding each generated token back as input for the next step (greedy, beam search, or sampling).

Exposure bias

Because training always sees gold prefixes but inference sees model-generated prefixes, errors can compound at test time. This mismatch is called exposure bias.

Where It Is Used

Variant	Attention	Typical use
Encoder-decoder	self + cross	translation, summarization, the original Transformer, T5
Encoder-only	bidirectional self	classification, embeddings (BERT)
Decoder-only	causal self	open-ended generation (GPT), see Decoder-Only Transformers

Translation is the canonical case: the source is fully known so a bidirectional encoder is ideal, while the target is produced left to right.

Code Sketch

import torch
import torch.nn as nn
from einops import rearrange
 
class EncoderDecoder(nn.Module):
    def __init__(self, d_model, n_heads, vocab):
        super().__init__()
        enc_layer = nn.TransformerEncoderLayer(d_model, n_heads, batch_first=True)
        dec_layer = nn.TransformerDecoderLayer(d_model, n_heads, batch_first=True)
        self.encoder = nn.TransformerEncoder(enc_layer, num_layers=6)
        self.decoder = nn.TransformerDecoder(dec_layer, num_layers=6)
        self.embed = nn.Embedding(vocab, d_model)
        self.head = nn.Linear(d_model, vocab)
 
    def forward(self, src, tgt):
        # src: (b, s), tgt: (b, t)  token ids
        memory = self.encoder(self.embed(src))          # cross-attn keys/values
        T = tgt.shape[1]
        causal = torch.triu(torch.ones(T, T, device=tgt.device) * float('-inf'), diagonal=1)
        dec = self.decoder(self.embed(tgt), memory, tgt_mask=causal)
        logits = self.head(dec)
        # (b, t, vocab) -> flatten for cross-entropy
        return rearrange(logits, 'b t v -> (b t) v')

The decoder consumes memory (the encoder output) through its internal cross-attention while the tgt_mask enforces causality on the target self-attention.

Related

The encoder-decoder is one of three families built on Transformers; it generalizes classical Seq2Seq RNNs, is powered by Self-Attention plus cross-attention, and contrasts with Decoder-Only Transformers which fold source and target into one causal stream.