Unlike in RNNs, transformers don’t natively have a notion of position, but we still want to have some sense of order (like in sentences, where the order of words does matter!)
RNNs worked like a state machine, reading the text one word at a time, keeping track of what they’d seen so far using a context vector. However, this approach is hard to parallelize, and the limited memory led to poor information retention/generation over long contexts.
For transformers, the representation of words and their position are separated into two distinct inputs, the token embedding and the positional embedding - this makes it far easier to parallelize.
We want positional embeddings to do a few things:
- Capture the distance between positions - words that are close together should have embeddings that are similar
- Be in the range of 0 and 1
- Work for any sequence length
- Be deterministic
Sinusoidal embeddings
- Sinusoidal functions bound our values between 0 and 1
- We can use the cosine distance to measure how far apart two positional embeddings are.
def positional_embedding(position: int, d_model: int = 1024):
i = np.arange(d_model)
angles = position / np.power(10000, (2 * (i // 2) / d_model))
angles[0::2] = np.sin(angles[0::2])
angles[1::2] = np.cos(angles[1::2])
return angles
Rotary Positional Embeddings
- Instead of adding/concatenating position info to word vectors (as with sinusoidal embeddings), RoPE rotates the token embeddings themselves so that the position information is embedded directly into the attention mechanism (the further the word is from the start, the more its rotated)
- Given a token embedding , we treat the vector as being made up of pairs:
- Each of these 2D pairs is then rotated by an angle that depends on the position of the token and the dimension of the pair.
- The rotation is done by using the standard 2D rotation matrix
- So when computing attention, instead of rotating just embeddings, both the queries and keys are rotated by the same function - directly incorporating relative position into the attention score.
