Researchers achieve full transformer performance with sparse tree-structured feed-forward layers activating less than 5% of parameters

Tree-structured conditional sparsity enables hard hierarchical routing in transformer MLPs without requiring separate router networks

Models with fewer than 5% of feed-forward units activated per token match dense baselines in autoregressive language modeling and question answering tasks

Approach scales successfully to models with over 1 billion parameters and works in zero-shot and few-shot settings

Emergent auto-pruning effect discovered during training where hard routing interacts with asymmetric nonlinearity, improving efficiency