A substrate-level fracture map of transformer inference.
This table decomposes a transformer inference step into atomic operations and tracks where strong semantic-manifold assumptions fail, fracture, or become only locally usable.
The purpose is not to say that transformer computation is invalid. The purpose is to show that useful transformer computation does not require a globally coherent semantic manifold.
Transformer inference alternates between identity-carrying substrate and fracture-producing transformation branches. Residual paths preserve enough structure for continuation, but they do not restore a global semantic manifold.
| Label | Meaning |
|---|---|
| STRUCTURAL | Discrete, identity-carrying, or operationally preserved substrate. |
| FRACTURED | Smooth semantic-manifold assumptions are broken by the operation. |
| MIXED | Residual recombination of identity-carrying state with transformed update. |
| VALIDATION | External operational test rather than a transformer sublayer. |
| Micro-Step / Layer | Mechanism | Operational State | Frac. ID | Status | Operational Analysis and Fracture Details |
|---|---|---|---|---|---|
| Phase 1: Injection — Discrete Substrate Enters the Model | |||||
| Token Selection | \( \text{argmax}(P) \) or \( x \sim P(x) \) | Discrete token event | 1 | STRUCTURAL | Fracture 1: Tokenization Quotient Break. The distribution collapses to an integer token. This is not smooth transport; it is a discrete restart of substrate. The next step begins from a selected token identity, not from a continuous semantic point. |
| Embedding Lookup | \( E[t] \in \mathbb{R}^d \) | Token-state handle | 2 | STRUCTURAL / FRACTURED | Fracture 2: Embedding Table Folding. The integer token is mapped into model state. Distinct tokens remain indexed, but training pressure can fold, compress, or interfere with token neighborhoods. The lookup is operationally useful, but it does not establish globally stable semantic coordinates. |
| Positional Injection | \( x + P_{pos} \) | Position-marked substrate | 3 | STRUCTURAL / FRACTURED | Fracture 3: Positional Phase Wrap. Position supplies order and prevents collapse into a bag of tokens. At the same time, rotary, periodic, or finite position mechanisms introduce seams and phase effects that do not correspond to semantic distance. |
| Residual Entry | \( x_{identity} \) | Identity-carrying state | – | STRUCTURAL | The residual stream carries persistence through the block. It is not a semantic manifold. It is the operational substrate that lets later updates be added without replacing the original token-state identity. |
| Phase 2: Attention Branch — Routing, Collapse, and Mixing | |||||
| Layer/RMS Norm | \( \frac{x - \mu}{\sigma} \) | Normalized surface | 9 | FRACTURED | Fracture 9: Normalization Geometry Rewriting. Magnitude is erased or rescaled. Dimensions become coupled through the normalization operation. Distances are recomputed in a new token-local geometry rather than preserved as a stable global metric. |
| Q/K/V Projections | \( x W_Q, x W_K, x W_V \) | Projected subspaces | 8 | FRACTURED | Fracture 8: Finite Precision Quantization. Linear projections operate after geometry has already been rewritten. In finite precision, traversal becomes lattice-constrained. Local linear structure may remain useful, but smooth global transport is not guaranteed. |
| Attention Scores | \( \frac{QK^T}{\sqrt{d_k}} \) | Scalar similarity field | 4, 10 | FRACTURED | The attention score operation scalarizes vector relations into pairwise weights. This is already a collapse of geometric structure into a routing field. Fracture 10 appears when instability produces Inf or NaN, creating hard representational holes. |
| Causal Masking | \( M_{ij} = -\infty \) if \( j > i \) | Lower-triangular boundary | – | FRACTURED | Causal masking imposes the arrow of time as a hard boundary condition. Future states are removed from the reachable attention field. This is operationally necessary, but it is not smooth manifold transport. |
| Softmax | \( \frac{e^{x_i}}{\sum e^{x_j}} \) | Probability simplex | 4 | FRACTURED | Fracture 4: Attention Softmax Saturation. Scores are forced onto a sum-to-one simplex. Dominance can collapse diversity, and gradients vanish for non-dominant routes. The result is a routing distribution, not a preserved semantic geometry. |
| Value Aggregation | \( \sum A \cdot V \) | Weighted mixture | 6 | FRACTURED | Fracture 6: KV-Cache Aliasing Pressure. Distinct histories and value states are mixed through weighted averaging. Useful information may be carried forward, but unique trajectory identity is compressed. Different histories can become operationally difficult or impossible to separate. |
| Output Projection | \( H_{attn} W_O \) | Reprojected update | – | FRACTURED | The attention result is mapped back into model dimension. This can reorient the update, but it cannot recover distinctions already destroyed by scalarization, masking, softmax saturation, or aggregation. |
| Phase 3: First Residual Integration — Partial Operational Restoration | |||||
| Residual Add 1 | \( x_{res} = x_{old} + x_{attn} \) | Identity plus attention update | 5 | MIXED | Fracture 5: Residual Dominance Shift. The identity-carrying stream is recombined with a fractured attention update. This restores operational continuity, not a global semantic manifold. The residual path preserves location while attention contributes a routed update. |
| Phase 4: FFN Branch — Expansion, Cut, and Refolding | |||||
| Layer/RMS Norm | \( \text{Norm}(x_{res}) \) | Renormalized state | 9 | FRACTURED | Fracture 9: Normalization Geometry Rewriting. The composite state is normalized again. Magnitude and metric relations are rewritten before the feed-forward branch operates. |
| Up-Projection | \( x W_{up} \) | Expanded workspace | – | FRACTURED | The state expands into a higher-dimensional workspace. This enables feature construction and separation, but it does not preserve original metric relations as a stable semantic surface. |
| Activation | \( \sigma(x) \) such as ReLU, GELU, or SwiGLU | Cut and warped workspace | 7 | FRACTURED | Fracture 7: MLP Activation Saturation. Activation functions cut, gate, flatten, or warp regions of the expanded workspace. These operations are useful for decision structure, but they violate smooth neighborhood transport. |
| Down-Projection | \( x W_{down} \) | Compressed update | 8 | FRACTURED | The branch compresses back to model dimension. Information shaped by activation survives as an update, but compression and finite precision prevent this from being a faithful manifold-preserving inverse. |
| Phase 5: Second Residual Integration — Continuity Without Global Repair | |||||
| Residual Add 2 | \( x_{res} = x_{old} + x_{ffn} \) | Identity plus FFN update | 5 | MIXED | Fracture 5: Residual Dominance Shift. The identity path stabilizes continuation while the FFN branch contributes transformed structure. This integration supports useful computation, but it does not certify a globally coherent semantic manifold. |
| Phase 6: Exit — Projection to Vocabulary and Discrete Re-entry | |||||
| Final Norm | \( \text{Norm}(x_{final}) \) | Final normalized state | 9 | FRACTURED | Fracture 9: Final Geometry Rewriting. Final normalization privileges directional comparison and erases magnitude information before vocabulary projection. |
| Unembed / Logits | \( x W_{vocab} \) | Vocabulary score field | 11 | FRACTURED | Fracture 11: Logit Rank Collapse. The model-dimensional state is projected into vocabulary space. The output scores occupy a constrained low-rank image rather than the full apparent vocabulary volume. |
| Final Softmax | \( \text{Softmax}(x) \) | Output simplex | 4 | FRACTURED | The final score field is smoothed into a probability distribution. The sampler then fractures this distribution back into a discrete token event, returning the process to Phase 1. |
| Test | Mechanism | Fracture | Status | Operational Meaning |
|---|---|---|---|---|
| Stress Testing | \( f(x+\epsilon) \) | 12 | VALIDATION | Fracture 12: Stress-Prompt Discontinuities. Small input changes should produce small, predictable output changes under a strong smooth-manifold hypothesis. In practice, tiny prompt edits can cross hidden fracture boundaries and trigger large behavioral jumps. |
| ID | Name | Definition |
|---|---|---|
| 1 | Tokenization Quotient Break | Discrete quotient singularities preclude a global continuous topology. |
| 2 | Embedding Table Folding | Training pressure can fold or compress token neighborhoods, undermining stable coordinate uniqueness. |
| 3 | Positional Phase Wrap | Phase seams and positional limits introduce coordinate singularities. |
| 4 | Attention Softmax Saturation | Degenerate response regimes make smooth transport unreliable. |
| 5 | Residual Dominance Shift | Nearby states may follow different effective compute paths as residual and update dominance shifts. |
| 6 | KV-Cache Aliasing | Trajectory injectivity breaks when distinct histories become operationally collapsed or inseparable. |
| 7 | MLP Activation Saturation | Local diffeomorphism fails as neighborhoods are cut, flattened, gated, or warped. |
| 8 | Finite Precision Quantization | Continuity is replaced by lattice-constrained numerical traversal. |
| 9 | Normalization Geometry Rewriting | Metric persistence is destroyed when distances and magnitudes are recomputed. |
| 10 | Undefined Numeric States | NaN or Inf create hard representational holes where total state coverage fails. |
| 11 | Logit Rank Collapse | The effective output dimension is constrained relative to apparent vocabulary space. |
| 12 | Stress-Prompt Discontinuities | Tiny prompt changes can produce large output or behavior jumps. |
Transformer inference survives because identity-carrying residual structure repeatedly recombines with fractured transformation branches.
This supports useful computation without rescuing the global semantic manifold hypothesis.
The sublayers do not alternate between invalid and valid semantics. They alternate between destructive transformation, operational recombination, and discrete re-entry.
What persists is not a smooth semantic manifold.
What persists is structured substrate under repeated repair.