rotary-embedding-torch

Why we included this project

Rotary position embeddings have become the default way modern transformer models encode sequence order, and this library packages that technique into a small, focused PyTorch utility you can drop into your own attention code. You instantiate a RotaryEmbedding, rotate your queries and keys, and keep your existing attention loop, which makes it easy to add RoPE without pulling in a full framework. It covers the edge cases that tend to bite in practice: offsetting positions when you have key/value caches at inference, axial embeddings for video or image transformers, and length extrapolation via XPos. There's also a fused Flash Attention path that applies the rotation in a single kernel, with a reference fallback when Triton isn't available. If you're writing custom attention or testing positional encoding variants, this is a well-scoped building block that won't drag in a heavyweight dependency.