# When Coordinates Aren’t Distances: Positional Encoding and Hierarchical Structure in Transformers
### The intuition from physics that reshapes how we think about language model position
—
For years, researchers working with transformer-based language models have accepted one foundational assumption: that position in a sequence is a single number — the token’s index — and that this number fully captures how “far apart” two tokens are from one another. RoPE rotates each channel pair by an angle proportional to this index. ALiBi adds a bias proportional to it. Different mechanisms, same object. They work because reading order determines token distances, and token distances are what attention cares about.
But what if that assumption is wrong? What if position in a language model is more like a coordinate in general relativity — a label that doesn’t directly tell you about the geometry of relationships between tokens?
This is the question that motivated a recent investigation into whether the hierarchical structure of text leaves a measurable fingerprint inside a transformer’s attention patterns. The answer is yes, and it points toward a deeper distinction between coordinates and distances that deserves attention from anyone who builds with or studies language models.
—
## Why Text Isn’t a Line
When we read text, we don’t experience it as a flat sequence. It has structure: documents contain paragraphs, paragraphs contain sentences, and sentences contain tokens. Two tokens that are five positions apart in reading order might sit in the same sentence, in different sentences within the same paragraph, or in entirely different paragraphs of a document.
A standard transformer’s positional encoding treats all five-position gaps as equivalent. It gives the model a single coordinate — a position on a line — and trusts that this is enough to represent where each token belongs. But that representation is lossy. The same linear sequence of tokens admits multiple valid hierarchical segmentations, and nothing in the positional encoding itself encodes which segmentation the model should use.
This raises an interesting possibility: what happens if we give the model explicit hierarchical coordinates and see whether attention responds differently to tokens based on their structural relationship, not just their distance along the sequence?
—
## A New Positional Encoding with Three Independent Coordinates
To test this, a hierarchical rotary positional encoding was designed — call it a multi-level position scheme — where paragraph index, sentence index, and token index each get their own independent set of rotational channels. Each level rotates a separate block of dimensions, and crucially, these blocks don’t interact with each other.
This design makes it possible to change one coordinate while holding everything else fixed. You can shift a token’s paragraph index without touching its sentence index, its token index, or anything else about the sequence. That separability is what makes clean experiments possible.
Several variants of the model were trained, all matched in parameter count, depth, learning schedule, and random seeds. Two control variants are especially important:
– **Random axial**: identical to the hierarchical model in every way except that the paragraph coordinate is a density-matched random label, resampled fresh at every training step. It can never carry real information about paragraph boundaries.
– **Periodic axial**: identical except that the paragraph coordinate follows a fixed, mechanical periodic grid at the same density as the true paragraph segmentation.
These controls isolate different aspects of what the paragraph channel might be doing. The random version tests whether simply having an extra coordinate matters at all. The periodic version tests whether a coordinate correlated with token distance — rather than one tracking genuine boundaries — produces the same effects.
—
## The Intervention: Inserting and Removing Boundaries
Once the models were trained, two targeted interventions were performed on their inputs:
– **Fake merge**: two adjacent real paragraphs were assigned the same paragraph index. The boundary between them vanished from the model’s perspective, even though the tokens and their order were unchanged.
– **Fake split**: two halves of a single real paragraph were given different paragraph indices. A boundary appeared where none existed before.
In both cases, the token-level identity, ordering, and pairwise token distances were held completely constant. The only thing that changed was the paragraph label. So if attention patterns shifted, that shift could only be attributed to the paragraph coordinate.
To measure the shift, attention was averaged across layers and heads, logged, and then fitted with a linear trend against token distance. The residual — the part of the attention pattern that token distance alone doesn’t explain — was what mattered.
The results were striking and consistent across three different corpora: WikiText-2, OpenWebText, and Python source code. Removing a real boundary lowered cross-paragraph attention. Inserting a fake boundary raised it. The two interventions pushed attention in opposite directions, proving together that the paragraph coordinate genuinely shapes how the model attends across structural boundaries.
—
## A Control That Complicates the Story
The random axial control also produced changes in cross-paragraph attention, in every corpus. The effect was weaker than in the hierarchical model, but it was real. So a change in cross-paragraph attention alone isn’t proof of paragraph structure — any channel with the right density can induce some shift. The real signal must be something more specific than just “having an extra coordinate.”
This is where the investigation deepened.
—
## Depth as the Signal
Instead of looking at whether attention changes when a boundary is inserted or removed, the researchers asked a subtler question: for every pair of tokens sitting in different paragraphs, how much *extra* suppression does their attention receive, beyond what token distance alone predicts? And then: how does that suppression change as the number of paragraph boundaries between them increases?
For the real hierarchical model, the answer formed a curve: suppression grew as paragraph displacement increased, reaching a peak around two to five boundaries apart, then decreased again. The shape looked like a well — a dip where the interaction is strongest. The depth of that well was called U-star.
For the random control, the well was shallower. And the key finding was that the depth behaved differently across corpora for the real model in a way it didn’t for the control. In two of three corpora, the real model compressed significantly deeper than the control. In the third, the difference was too small to resolve.
The control’s depths sat in a narrow band regardless of corpus. The real model’s depths varied widely — a range more than four times larger. That corpus-sensitivity is what suggested the depth wasn’t just an artifact of coordinate density. Something about the real paragraph structure was being read out.
—
## Causally Testing the Effect
Correlation isn’t causation, so a within-corpus test was designed. The order of paragraphs in each document was shuffled — same content, same lengths, same number of paragraphs, only the order changed.
If depth reflects genuine paragraph structure, shuffling should move it. If depth is just an artifact of the channel’s statistical properties, shuffling should do nothing.
The results confirmed the causal story. For the hierarchical model, depth changed in every corpus after shuffling — sometimes getting shallower, sometimes deeper, depending on the corpus, but always shifting in a reproducible way. For the random axial model, depth didn’t change at all. Every confidence interval included zero.
The shuffle changed the input equally for both models. What differed was what each model had learned to do with the paragraph coordinate. The hierarchical model, trained on real paragraph order, had something to lose when that order was disturbed. The random model had not.
—
## What Corpus Statistics Can and Cannot Explain
Efforts were made to explain the corpus-dependent depth using only properties of the raw text — no model involved. Three families of statistics were examined: lexical persistence, paragraph length distributions, and embedding-based coherence measures. Eight quantities in total.
None fully reproduced the observed ordering of U-star across corpora. Lexical persistence got one corpus right but swapped the other two. Paragraph length swapped a different pair. Embedding coherence came closest — getting two out of three corpus comparisons right — but left one case unresolved.
This is a genuinely negative result worth reporting honestly. The corpus-dependent depth is real and causally tied to paragraph structure. But *what specific property of a corpus* determines how deep the well is remains an open question, and the author welcomes discussion on the topic.
—
## The Honest Caveats
Two important limitations should be front and center.
First, one corpus — OpenWebText — showed no depth gap between the real model and the control at the tested scale. The models were statistically indistinguishable there on that particular comparison. This doesn’t mean the causal effect vanishes; the shuffle test still distinguished real from random in that corpus. It means the depth comparison, in particular, wasn’t resolvable with the available data.
Second, the models used were eight layers deep with 512 dimensions, trained across three seeds and three corpora. The phenomenon is reproducible at this scale. Whether it persists at production scale with billions of parameters is not something this work answers.
—
## The Core Insight: Coordinates Are Not Distances
The conceptual payoff of this work is a reframing that draws on an intuition from general relativity.
In Einstein’s theory, coordinates label events in spacetime, but they don’t tell you about distances. The distance between two events depends on the metric — a function of position that varies from place to place. Two events with the same coordinate difference can have very different proper distances, depending on where they sit in the geometry.
The parallel inside a transformer is structural, not physical. The token index is a coordinate — exact, linear, and unambiguous for the one-dimensional sequence. But the “distance” that attention operates over isn’t the same thing as coordinate difference. Two tokens separated by the same number of positions attract different amounts of attention depending on their hierarchical relationship.
In the language of physics: the positional coordinate is not the metric. The metric depends on the coordinate *and* on the structure of the sequence. What was measured here is a specific piece of that dependence — how attention weight varies with paragraph displacement when token distance is held constant.
The strongest claim that can be defended is this: attention exhibits a structured, coordinate-dependent response field over hierarchical textual position, whose qualitative shape is consistent with a metric-like dependence on structural separation. The well is real in the sense that it is reproducible across corpora and causally tied to paragraph structure. It is not real in the sense of being a derived geometric object.
The honest analogy is narrower than “transformers are relativistic.” It is this: **the coordinate that standard positional encodings supply is not the same object as the distance that attention actually computes over.**
—
## What Comes Next
Two directions stand out as natural follow-ups. The first is scale: does the depth signature persist in models with a billion or more parameters, or was it a feature of the small architectures tested here? The second is more corpora: three shows the pattern exists and is causally connected to paragraph structure, but it is not enough to determine *what* in a corpus sets the depth of the well.
The paper has been made publicly available, and the author has expressed particular interest in feedback on the OpenWebText null result, where the depth comparison between real and random models was inconclusive.
—
## Frequently Asked Questions
**Q: What is the difference between a coordinate and a distance in this context?**
A coordinate is simply a label — in this case, the token’s position index in the reading order. A distance is what attention actually computes: a measure of how much one token should attend to another. The core finding is that these two things are not the same in transformers. Two tokens at the same coordinate distance can have very different attention weights depending on their structural relationship in the text hierarchy.
**Q: Why does the paragraph coordinate matter if token distance already captures “how far apart” tokens are?**
Because token distance alone treats all five-token gaps as equivalent, regardless of whether those five tokens are all in the same sentence, span multiple sentences in one paragraph, or sit in different paragraphs of a document. The paragraph coordinate gives the model additional structural information that changes how it weighs attention across boundaries.
**Q: What does “fake merge” and “fake split” actually do in practice?**
Fake merge takes two adjacent paragraphs that the model has learned to treat as separate units and assigns them the same paragraph label, erasing the boundary from the model’s input. Fake split takes one paragraph and assigns its two halves different paragraph labels, creating a boundary where none exists in the original text. In both cases, every token, every order, and every pairwise token distance stays identical — only the paragraph label changes.
**Q: Why did the random axial control still show some effect?**
Because simply adding an extra channel with a certain statistical density to the positional encoding can influence attention patterns. The random coordinate is not carrying real structural information, but having a channel that varies at a particular density interacts with the model’s attention mechanism in ways that produce measurable changes. The real model shows a much stronger and more structured effect, and its depth is causally sensitive to paragraph order — something the random model’s depth is not.
**Q: What does it mean that depth is “corpus-sensitive”?**
The depth of the compression well — the most negative point on the curve of attention suppression versus paragraph displacement — varies in magnitude across different text corpora for the hierarchical model, but stays nearly constant across corpora for the random control. This suggests that the real model is picking up something about the actual paragraph structure of each corpus, rather than just responding to a generic statistical property of having a paragraph coordinate channel.
**Q: Does this mean transformers understand text the way humans do?**
Not necessarily. What the evidence shows is that transformers develop attention patterns that respond to hierarchical textual structure in a systematic and measurable way. This is not the same as claiming human-like comprehension. However, it does suggest that the standard positional encoding is missing information that the model finds useful, and that giving the model explicit structural coordinates changes how it processes text.
**Q: Is the paper claiming that transformers are “relativistic”?**
No. The paper uses general relativity as an analogy to illustrate a conceptual point: that coordinates and distances are different objects, and that conflating them is lossy. The actual finding is narrower — that attention computes over a distance that depends on hierarchical position in a way that goes beyond simple token-index coordinates.
—
## Conclusion
The relationship between position and distance in language models is more nuanced than the standard positional encoding paradigm assumes. Text is not a flat sequence — it is a nested hierarchy, and transformers appear to exploit that hierarchy in their attention patterns in ways that a single coordinate cannot capture. By introducing a hierarchical positional encoding and carefully isolating the effect of paragraph structure through targeted interventions, the research demonstrates that attention fields over text have a structured, geometry-like dependence on hierarchical position.
This does not mean transformers have discovered a new physics. It means that the tools we use to represent position in these models — the very coordinates we assume are sufficient — are actually losing information about the structure of the data they process. The gap between coordinate and distance, familiar from general relativity, turns out to have a useful analogue inside language models.
Whether this insight leads to better positional encodings, more efficient architectures, or simply a deeper understanding of what transformers have learned about text structure remains an open and exciting question.
—
Thank you for reading
