Hand-coded AI weights match trained models' scaling efficiency

The researchers constructed weights by hand for single-layer MLPs tasked with memorizing labels associated with two-token input sequences. Rather than using standard gradient-based training, they directly coded the weight values, then measured how many facts each model could memorize at 90% accuracy. Scaling the models up by increasing parameter count revealed a roughly linear relationship between model size and memorization capacity—the same linear scaling observed in models trained conventionally. This parallel scaling behavior is noteworthy because it shows that the efficiency of the scaling relationship is not unique to learned weights but emerges from the architecture itself. Yet the hand-coded approach did not fully match trained models: the scaling prefactor—the constant that multiplies the linear term—fell short of the trained models' prefactor by a specific factor (the article's rendering of this factor is incomplete in the available text). This gap indicates that while architecture determines the form of scaling (linear in this case), the training process optimizes weights in ways that improve the absolute capacity within that scaling regime. The finding suggests a decomposition of neural network efficiency: one part stems from structural properties of the architecture, while another part emerges from optimization during training.

The experiment addresses a foundational question in deep learning: whether efficient scaling emerges from network architecture alone or requires the optimization process that gradient-based training provides. By hand-coding weights rather than training them, the researchers isolate the contribution of architecture structure to memorization capacity. The result—that scaling behavior matches between hand-coded and trained models—suggests that the linear relationship between parameters and memorized facts is largely determined by the MLP structure itself, not by the particulars of the training algorithm. However, the gap in the scaling prefactor reveals that trained models extract additional efficiency from optimization, learning weight configurations that trained models can achieve but hand-coding cannot easily reproduce. This distinction between structural scaling and optimization-driven improvement clarifies what gradient descent adds beyond what is already implicit in the network architecture.

No comments yet. Be the first to share your thoughts!