Multiverse's CBO beats rivals by almost 23 points on MMLU

Multiverse Computing's paper starts from a blunt but effective idea: delete whole transformer blocks to make a large language model shorter and therefore faster, with predictable inference speedups that stack on top of quantization and low-rank compression. The hard part has always been deciding which blocks to cut. Most existing methods score each block on its own, using magnitude, sensitivity, or block influence heuristics, and some only ever remove a single consecutive run of blocks. The paper argues that both shortcuts throw away quality because removing one block depends on which others you remove alongside it, making this a combinatorial problem rather than a ranking problem.

The authors take that interaction seriously by attaching a binary variable to each block and doing a second-order Taylor expansion of the model's loss, which produces an approximate Hessian. Its diagonal captures how much each block matters alone, and its off-diagonal entries capture the pairwise couplings between blocks. That turns block selection into a constrained binary optimization problem, physically an Ising glass with a fixed number of removed blocks. The key practical property is cost: the Hessian is computed once from a small calibration dataset, and after that any candidate configuration can be scored with a single cheap energy calculation, so the same Hessian can be reused for many different compression targets.

For tractable cases the method brute-forces up to tens of billions of configurations on a single GPU, and for harder cases it hands the same problem in QUBO form to classical, quantum, and quantum-inspired solvers, with an open-source tabu solver reliably reaching the lowest-energy states in seconds. The paper's most striking practical finding is that the single lowest-energy state is not always the best model. For Llama-3.1-8B-Instruct at 16/32 blocks removed, the 17th excited state proposes removing a block near the beginning of the model and, after light retraining, outperforms the ground state across several benchmarks. Whether this holds across other architectures and compression levels is likely the test of how broadly the approach generalizes beyond the dense and hybrid models tested here.