Rotary Position Embedding (RoPE)

Attention does not know the order of tokens, so a Transformer needs some way to tell it where each token sits. The original Transformer adds a sinusoidal vector to the input embeddings. RoPE (Rotary Position Embedding) takes a different route: instead of adding anything, it rotates the query and key vectors by an angle that depends on their position. As a result, the attention score between two tokens depends on how far apart they are, which is exactly the information language needs. This page covers the motivation, the rotation idea, the formulas, the role of different frequencies, the relative-distance property, long-term decay, and a small implementation.

In attention, the influence of one token on another is measured by the score matrix \(QK^T\). Sinusoidal positional encoding adds position information only once, to the input embeddings, so it reaches the scores indirectly through the learned projections and many layers. Consider "The dog chased another dog": both "dog" tokens have the same embedding, and only the added position vector tells them apart. RoPE asks a more direct question: can the queries and keys, which produce the scores, carry the position themselves?

Sentence : "The dog chased another dog"      (5 tokens, vectors of size 4)
Q  : 5 x 4     K^T : 4 x 5     Scores = Q K^T : 5 x 5

Without position information:
  "dog" (token 2) and "dog" (token 5) have identical vectors,
  so they would receive identical attention scores.
Goal: make the score between two tokens depend on their distance.
    

RoPE treats a vector as an arrow in a plane and turns it by an angle proportional to the token's position. The query at position \(m\) is rotated by \(m\theta\) and the key at position \(n\) is rotated by \(n\theta\). Notice that nothing is added to the vector: positional information is included only by rotating it, which keeps its length unchanged. After the rotation, the angle between the query and the key is \((n-m)\theta\), so their dot product depends only on the distance \(n-m\) and not on the absolute positions.

theta = 25 degrees, query at position m = 2, key at position n = 5

query is rotated by m * theta = 50 degrees
key   is rotated by n * theta = 125 degrees
angle between them           = 125 - 50 = 75 degrees = (n - m) * theta

Moving both tokens 10 positions to the right changes both angles,
but the angle between them stays 75 degrees.
    

The rotation by an angle \(m\theta\) is done with the standard 2D rotation matrix:

\(R_{\theta,m} = \begin{pmatrix} \cos(m\theta) & -\sin(m\theta) \\ \sin(m\theta) & \cos(m\theta) \end{pmatrix}\)

Multiplying a 2D vector \((x_1, x_2)\) by this matrix gives the rotated vector \((x_1', x_2')\) with \(x_1' = x_1\cos(m\theta) - x_2\sin(m\theta)\) and \(x_2' = x_1\sin(m\theta) + x_2\cos(m\theta)\). Some sources write the matrix with the signs of the sine terms swapped, which just means the rotation is applied in the opposite direction; the idea is the same. Position \(m=0\) means no rotation at all.

theta = 30 degrees, vector q = (2, 0)   (length 2)

position m = 0 : angle   0 degrees -> (2.000, 0.000)
position m = 1 : angle  30 degrees -> (1.732, 1.000)
position m = 2 : angle  60 degrees -> (1.000, 1.732)

Each result is the same arrow turned further around; the length stays 2.
    

Real query and key vectors have many dimensions (for example 64 or 128), but a rotation is easiest to describe in a plane. RoPE therefore splits the \(d\) dimensions into \(d/2\) consecutive pairs and rotates each pair in its own plane, using its own angle \(\theta_i\). The full rotation matrix is block-diagonal, with the 2 × 2 rotation of each pair on the diagonal and zeros elsewhere:

\(R_{\theta,m} = \begin{pmatrix} \text{block}_1 & 0 & \cdots & 0 \\ 0 & \text{block}_2 & \ddots & \vdots \\ \vdots & \ddots & \ddots & 0 \\ 0 & \cdots & 0 & \text{block}_{d/2} \end{pmatrix}, \quad \text{block}_i = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \\ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix}\)

Some write the angle as \(\omega_i = 1/10000^{2i/d}\) with \(i\) starting at 0; it is the same set of values. In practice the big matrix is never built: each pair is rotated directly, so the cost is small and RoPE adds no learnable parameters.

d = 4  ->  2 pairs, theta_1 = 1, theta_2 = 0.01
Vector x = [1, 0, 1, 0]

position 0 : [ 1.000, 0.000, 1.000, 0.000]   (no rotation)
position 1 : [ 0.540, 0.841, 1.000, 0.010]   (pair 1 turned 1 rad, pair 2 turned 0.01 rad)
position 2 : [-0.416, 0.909, 1.000, 0.020]
    

Each pair has its own rotation speed, set by \(\theta_i\). The rotation angle of pair \(i\) at position \(p\) is \(p \cdot \theta_i\), which grows with the position. Three consequences follow. (1) Larger position differences lead to larger rotations, so nearby tokens get similar rotations while distant tokens get different ones. (2) Low-index pairs change quickly with position (high frequency), and high-index pairs change slowly (low frequency). (3) The fast pairs capture small shifts, while the slow pairs keep track of long distances, much like the hands of a clock.

Compare "I just told her the truth"  vs  "I told just her the truth"

Only the word "told" moves by one position. A fast pair (large theta)
rotates noticeably for a 1-position shift, so the model can see the
difference. A slow pair barely changes, so it cannot tell these two
sentences apart but is useful for relations over long distances.
    

Take the token vectors \(x_m\) and \(x_n\), project them with \(W_q\) and \(W_k\), and rotate the results by their positions. The rotated query is \(x_m W_q R_{\theta,m}^T\) and the rotated key is \(x_n W_k R_{\theta,n}^T\). When their dot product is taken, the two rotations combine into one, because \(R_{\theta,m}^T R_{\theta,n} = R_{\theta,n-m}\). The absolute positions disappear and only the relative distance \(n-m\) remains:

\(q_m k_n^T = x_m W_q R_{\theta,m}^T R_{\theta,n} W_k^T x_n^T = x_m W_q R_{\theta,n-m} W_k^T x_n^T\)

So the model learns patterns such as "the word two places before me" in a way that works at any position in the text.

2D case, theta = 30 degrees, query = key = (1, 0) before rotation.
The dot product of the rotated vectors is cos((n - m) * theta):

m = 1, n = 3  (distance 2) -> cos(60 degrees) = 0.50
m = 5, n = 7  (distance 2) -> cos(60 degrees) = 0.50   (same distance, same score)
m = 2, n = 3  (distance 1) -> cos(30 degrees) = 0.87   (closer, higher score)
    

The RoFormer paper shows that the relative upper bound of the attention score \(A(q_m k_n^T)\) decreases as the distance \(|m-n|\) grows. The reason is the mix of frequencies: for a small distance all pairs are rotated by similar amounts and their contributions add up, but for a large distance the fast-rotating pairs point in different directions and partly cancel each other. This matches the intuition that tokens close to each other are usually more relevant than far-away ones. It is a bound, not a rule, so the model can still attend strongly to a distant token when the content demands it.

Relative upper bound (d = 256, scaled so that distance 0 = 100%):

distance   0 : 100%
distance  50 :  about 35%
distance 150 :  about 23%
distance 255 :  about 19%

A token 5 positions away is influenced much more than one 200 positions away.
    

RoPE is applied after the linear projections that create Q and K, inside every attention layer. For each token, the vector is split into pairs, and each pair is rotated by (position × \(\theta_i\)) using the element-wise formulas from section 3. The value vectors are not rotated, because positions only need to influence which tokens are attended to, not the content that is passed on. The code below rotates one vector and then checks the relative-distance property from section 6.

import numpy as np

def rope(x: np.ndarray, pos: int, base: float = 10000.0) -> np.ndarray:
    d = len(x)                                   # d must be even
    i = np.arange(d // 2)
    theta = base ** (-2 * i / d)                 # one angle per pair
    angle = pos * theta                          # rotation angle of each pair
    x1, x2 = x[0::2], x[1::2]                    # split into pairs
    out = np.empty_like(x, dtype=float)
    out[0::2] = x1 * np.cos(angle) - x2 * np.sin(angle)
    out[1::2] = x1 * np.sin(angle) + x2 * np.cos(angle)
    return out

# Relative-distance check: the score only depends on n - m
rng = np.random.default_rng(0)
q, k = rng.normal(size=8), rng.normal(size=8)
for m, n in [(1, 3), (5, 7), (10, 12), (2, 3)]:
    print(m, n, round(float(rope(q, m) @ rope(k, n)), 4))
    

Output of the check above:
1 3  -1.3804
5 7  -1.3804
10 12 -1.3804      <- distance 2 every time: identical scores
2 3  -1.3091       <- distance 1: a different score