Hand-coded weights match trained models' scaling, but fall short in efficiency

Researchers conducted an experiment to understand the relationship between neural network architecture, parameter count, and memorization capacity. They hand-coded the weights of single-layer MLPs—networks with one hidden layer and an input-output structure—rather than training them with gradient descent. The task was narrow and well-defined: each model had to memorize correct labels for two-token input sequences.

The key finding was that the number of facts the hand-coded models could memorize with 90% accuracy scaled roughly linearly with the models' parameter count. This scaling behavior matched what is observed in trained models using the same architecture, suggesting that the linear relationship is not a quirk of the training process but rather a fundamental property of how this class of networks can encode information.

However, the researchers also found that their hand-coded models underperformed trained models in absolute efficiency terms. While both scaled linearly with parameters, the hand-coded models' scaling prefactor—the multiplicative constant that determines how many facts can be memorized per parameter—fell short of trained models' by a measurable factor. This gap reveals that although architecture determines the form of scaling (linear, in this case), the training process discovers more efficient weight configurations than manual design can achieve.

The research addresses a fundamental question in neural network science: whether the scaling laws observed in trained models are intrinsic to architecture or artifacts of the training process. By constructing weights by hand rather than through gradient descent, the researchers isolated the architectural contribution to scaling behavior. The finding that both approaches scale linearly suggests the relationship is robust—a property of how MLPs can theoretically encode information—rather than an accident of learning dynamics.

Yet the gap in efficiency between hand-coded and trained weights indicates that training algorithms discover weight configurations that are substantially better at using available parameters. This gap points to a real constraint: even when we understand the scaling law, realizing it efficiently still requires the optimization process that training provides. The work thus carves out a middle ground: architecture determines the form of scaling, but training determines how well that scaling is achieved.

No comments yet. Be the first to share your thoughts!