Researcher builds compiler turning computation graphs into transformer weights—no training needed

The researcher set out to answer a specific question about transformer expressivity: what algorithms can a transformer actually execute, independent of what it learns during training? To explore this, they built a compiler that takes a user-defined computation graph in ordinary Python and produces a complete set of transformer weights that will execute that graph.

The compiler outputs a standard Phi-3-architecture checkpoint—a widely used transformer format—that loads directly in Hugging Face's transformers library without requiring custom code or the trust_remote_code flag. Critically, the entire pipeline involves zero training: the weights are constructed directly from the computation graph specification. This means the transformer's behavior is deterministic and fully traceable to the input algorithm, rather than emerging from a learning process.

The approach is not entirely novel. Prior work, notably the RASP language and the Tracr compiler, has explored similar ideas: RASP defines a language whose primitives map onto transformer sublayers (attention, feedforward, etc.), and Tracr compiles RASP programs into actual transformer weights. However, those systems require users to learn a domain-specific language and use custom loading code to run the resulting weights. The new compiler addresses both limitations: it accepts computation graphs in ordinary Python, making it more accessible to developers familiar with standard programming, and it targets a stock transformer architecture, eliminating friction in deployment.

The researcher has published a detailed technical write-up explaining the construction methods and released a public repository with twelve runnable examples, providing a foundation for others to understand and extend the approach.

The work builds on earlier ideas about hand-building transformer weights. RASP (a prior system) defines a language whose primitives map onto transformer sublayers, and Tracr compiles RASP programs into actual weights. However, both systems required users to work within domain-specific languages and custom loading infrastructure. This new compiler addresses two specific gaps: it allows computation graphs to be expressed in standard Python (lowering the barrier for users unfamiliar with specialized languages) and targets a stock architecture (Phi-3), eliminating the need for custom code or trust_remote_code flags when loading the result.

The significance lies in the question the author is pursuing: what can transformers express as algorithms, separate from what they can learn through training? By compiling a computation graph directly into weights with zero training, the work provides a concrete answer for certain classes of problems—demonstrating that some computations can be embedded in a transformer's structure by design. This has implications for understanding transformer expressivity and potentially for interpretability research, since the relationship between input computation and output weights is fully explicit.

No comments yet. Be the first to share your thoughts!