GPT-2's 32,070-token vocabulary mapped in interactive hyperbolic space

GPT-2-small's vocabulary consists of 32,070 tokens—the discrete units of text the model reads and generates. Each token is represented internally as a vector of numbers (an embedding), and these embeddings capture semantic and syntactic relationships: similar tokens cluster together in the embedding space. A researcher analyzed this structure and discovered it has a clear forest topology: one main tree containing about 2,300 tokens (likely common words and word pieces), several hundred smaller trees of specialized vocabulary clusters, and approximately 6,700 isolated tokens with no close neighbors. In traditional flat (Euclidean) space, tree structures suffer from distortion and inefficiency—branches at different depths require vastly different scaling to display legibly. Hyperbolic geometry avoids this problem. In hyperbolic space, the amount of surface area (or volume in higher dimensions) grows exponentially as you move outward, which is precisely the property that allows trees to be embedded efficiently. The creator used the Poincaré ball model, a standard mathematical representation where hyperbolic space is mapped into a unit ball, and populated it with GPT-2's actual token embeddings. The resulting visualization is fully interactive and runs directly in a web browser on mobile devices. Users can drag their finger to rotate the view, pinch to zoom, and tap any token to perform a Möbius translation—a transformation that smoothly brings that token to the center of view while the rest of the space reorganizes around it. This is the natural, distortion-minimizing way to navigate hyperbolic geometry. Tapping neighboring tokens allows continuous exploration of the vocabulary's structure. The visualization uses only GPT-2-small's raw embeddings with no additional processing, making it a faithful window into how the model encodes its discrete vocabulary.

This visualization project demonstrates a principled approach to understanding how large language models organize their vocabulary at scale. Rather than imposing a conventional 2D projection, the creator recognized that GPT-2's token embeddings naturally encode a hierarchical, tree-like structure—confirmed by the presence of one dominant cluster (around 2,300 tokens), numerous smaller family trees, and thousands of isolated tokens. Hyperbolic geometry provides a mathematically natural container for such structures because, unlike Euclidean space, it offers exponentially more room as you move away from the origin, making hierarchies and trees fit without distortion. By embedding this structure in a Poincaré ball (the standard computational model of hyperbolic geometry), the visualization becomes both mathematically faithful to the data's intrinsic shape and practically interactive on consumer hardware. The mobile-first implementation with touch-native controls (Möbius translation for navigation) makes this a rare example of hyperscale model internals presented in a format users can actually explore.

No comments yet. Be the first to share your thoughts!