On November 15, 2026, Sebastian Raschka published a long-form tutorial on how modern LLMs represent token positions — from GPT-2's learned tables through RoPE math to NoPE-only hybrids and Inkling's learned relative bias. If you build with open weights, fine-tune, or debug long-context failures, this is the missing bridge between our transformer architecture primer and per-model cards that only say "uses RoPE" in a bullet.
Raschka's piece is intentionally dense — figures, equations, and a runnable PyTorch RoPE implementation. This explainx.ai summary extracts what changed in 2026 open models and what practitioners should verify, while pointing you to the original article for proofs and code.
TL;DR: schemes and when they matter
| Scheme | Where position enters | Typical models | Builder takeaway |
|---|---|---|---|
| Learned absolute | Added to token embeddings pre-block | GPT-2 lineage | Poor length extrapolation beyond train context |
| RoPE (full) | Rotation on Q/K in attention | Most open LLMs since LLaMA | Default; watch rope_theta / scaling at long context |
| Partial RoPE | Rotate subset of head dims | MiniMax-M3 (64 of 128 dims) | Saves small table memory; mixed benchmark impact |
| NoPE layers | No explicit encoding; causal mask only | SmolLM3 pattern (every 4th block) | Rising in 2026 gallery per Raschka |
| Learned relative bias | Bias added to attention logits | Inkling, historical T5 | Different inductive bias for long gaps |
Why position encoding still confuses builders
Self-attention without position information treats token pairs as unordered bags. Swap "red fox" and "gray fox" and you change the story, but not the raw ID embeddings for repeated words like "fox."
Raschka's opening toy example walks two sentences that differ only by word order. He notes causal masking already gives some order signal — a token sees only prior tokens — but not fine-grained distance: the mask cannot distinguish one step from ten steps apart. That gap is why explicit schemes remain common even in autoregressive decoders.
Our earlier transformer guide summarized learned embeddings versus RoPE at a high level. Raschka's November piece finally supplies mechanistic code and a 2026 gallery bar chart showing RoPE-everywhere declining as hybrids rise.
Wave above a ruler illustrating rotary positional encoding along a token sequence
RoPE in plain language
Rotary positional embedding (RoPE) multiplies pairs of features in queries and keys by position-dependent rotations. Dot products then depend on relative distance, which is what you want when attention scores must know how far apart two tokens are.
Raschka implements split-half pairing (modern default) versus interleaved pairing (original paper / early LLaMA). He shows with torch.testing.assert_close that layouts are equivalent when you reorder features consistently — important when you port weights between codebases.
For implementers, the actionable detail is not the trigonometry but consistency: inference engines must apply the same convention, base frequency, and scaling rules the checkpoint trained with. That is where llama.cpp and ggml releases matter when you stretch context with YaRN or similar.
Partial RoPE and memory math
MiniMax-M3 rotates half of each 128-dim head (64 rotary dims). Raschka calculates BF16 table savings on a 1M-token context — on the order of 128 MiB for cosine/sine tables — while noting KV cache still stores full keys and values.
He cites the Fractional Rotation, Full Potential? paper finding ~10% rotation can match full RoPE on some small-model trainings, but full RoPE wins on certain evals. His editorial verdict: partial RoPE is often not worth the complexity unless you are squeezing last MiB at megacontext.
NoPE layers and SmolLM3's pattern
The NoPE paper showed causal attention alone can encode position via features that average differently as prefix length grows. Raschka reproduces the toy "one hot The token" example where uniform attention yields 1/prefix_length statistics downstream layers can read.
At scale, results are mixed: tiny models generalize length on synthetic tasks; 1.3B code models drift at extreme lengths. In 2026 open weights, Raschka highlights SmolLM3 (3B) applying RoPE in three layers and skipping every fourth block — authors reported improved long-context behavior.
That connects to explainx.ai's Kimi K3 architecture notes, where Raschka previously documented NoPE everywhere in a frontier open stack. The lesson: do not assume RoPE when you pick a fine-tune base — read the gallery card.
Attention mechanism spotlight illustrating how RoPE changes query-key geometry in transformers
Inkling and learned relative bias
Before the gallery summary, Raschka covers Inkling's learned relative position bias — add a term to scaled Q·K based on hidden state and token distance — and links it to T5's simpler bucketing. Thinking Machines' Inkling post on explainx.ai already noted relative embeddings instead of RoPE at the announcement layer; Raschka's section supplies the attention-diagram intuition.
What 2026 open models are doing (per gallery)
Section 6.4 aggregates Raschka's LLM Architecture Gallery snapshots:
- RoPE in every softmax layer remains the plurality but share fell from 2025 into 2026.
- Mixed RoPE + NoPE blocks grew — still a minority but directionally interesting.
- NoPE-only entries in the snapshot were recurrent hybrids without classic softmax everywhere.
- Other includes architectures like xLSTM (no softmax attention) and Inkling (learned bias).
For builders choosing a base model in November 2026, treat the chart as release-bias data, not a performance ranking: galleries track what shipped, not what won every benchmark.
What people search after seeing "RoPE explained"
Search logs cluster on "rotary positional embeddings rope explained" and "positional embedding vs rope" — both ask the same practical question: If I already know transformers, what do I change in code?
Raschka's answer: stop adding position vectors before block 0 unless you are debugging GPT-2; instead trace where rotations hook into Q/K, whether partial dims are frozen, and whether later NoPE blocks still inherit position from earlier rotations.
Secondary searches on "llm meaning" are noise for this article — ignore them in implementation work.
Split-half versus interleaved RoPE in production code
Raschka spends meaningful space on a detail most blogs skip: original RoPE interleaved feature pairs, while most 2026 implementations use split-half pairing because vectorized sine/cosine ops are simpler. Weight ports break when frameworks disagree on ordering — not when the math differs.
If you fine-tune with Hugging Face Transformers then deploy with a C++ runtime, verify the export script's rope_scaling JSON matches both sides. Our local GLM tensor-parallel write-up is a reminder that deployment blogs must mention implementation parity, not just peak tokens/sec.
From GPT-2 learned vectors to rotation inside attention
Section 2 of Raschka's article revisits learned absolute tables — the approach in GPT-2 and BERT. They are expressive within train context but famously weak at extrapolation. Section 3–5 build RoPE intuition with frequency bases, wavelength metaphors, and the PyTorch rope_split_halves function.
Readers who only ever invoked AutoModel.from_pretrained still benefit: when a model card says "2048 train context, 128k inference," the magic is usually RoPE scaling plus KV tricks, not a bigger embedding table.
Causal masking alone is not enough — but NoPE exploits it
Raschka's NoPE discussion is careful not to overclaim. Causal masks provide prefix statistics exploitable by later layers — his uniform-attention toy model shows how a feature counting The yields 1/n signals. Real models do not use uniform attention, but the ablations in the NoPE paper convinced authors that some layers can drop explicit encodings, especially when earlier layers already rotated Q/K.
SmolLM3's every fourth block pattern is the pragmatic compromise: keep most layers familiar to tooling, experiment where long-context evals demanded it.
Gallery methodology caveat
The bar chart in Section 6.4 counts architecture cards in Raschka's gallery, not market share by download volume. A recurrent hybrid with NoPE-only softmax layers counts once even if its checkpoint is niche. Use the gallery to audit your own assumptions, not to declare RoPE "dead."
How to use this when debugging long context
| Symptom | Check |
|---|---|
| Garbage after 32k tokens | RoPE scaling / YaRN params vs model card |
| Divergence after quantizing KV | Partial RoPE dims and cache layout |
| Fine-tune collapse on order-sensitive task | Accidental NoPE block in base model |
| Port weights between frameworks | Split-half vs interleaved pairing |
Pair those checks with hardware context from local inference guides only after the math matches — faster GPUs do not fix wrong rope_theta.
Related reading
- What is a transformer? — attention foundations
- Kimi K3 architecture and NoPE — prior Raschka coverage
- Inkling open weights — relative bias in production model
- Vals AI eval transparency — how labs publish benchmark context alongside architecture posts
- GLM local tensor parallel — when architecture details meet deployment
Primary source: Understanding Positional Embeddings in Modern LLMs — Sebastian Raschka, Nov 15, 2026
Classroom use and follow-on articles
Raschka notes the piece is a foundation for an upcoming LLM architecture trends article — expect cross-links when that publishes. Educators can assign Section 1–5 for graduate labs and Section 6 for seminar discussion on inductive biases. Substack supporters get gallery image exports; open readers still benefit from the public figures embedded in the web view.
If you teach from Build a Large Language Model (From Scratch), this article is the positional-encoding companion chapter the book's attention module points toward.
Gallery counts and model examples reflect Raschka's article as published November 15, 2026.