DiScoFormer: One transformer estimates density and score without retraining

Key takeaway

• Researchers have introduced DiScoFormer, a transformer model that estimates both density and score—two key quantities describing how data is distributed—in a single pass without retraining on new datasets.

• The model significantly outperforms classical methods in high dimensions and has broad applications across generative AI, Bayesian inference, and scientific computing, potentially reducing retraining costs across these fields.

3 Key Points

FAQ

How does DiScoFormer perform compared to classical kernel density estimation?

In 100 dimensions, DiScoFormer cuts score error by about 6.5x and density error by more than 37x against the best hand-tuned kernel density estimation, and continues improving as you add samples while the classical method runs out of memory. Kernel density estimation's main remaining advantage is speed, especially with small datasets.

What data was used to train DiScoFormer?

The model was trained on Gaussian Mixture Models (GMMs), which are universal density approximators and have closed-form densities and scores that serve as exact supervision targets. A new GMM was drawn for every batch, giving the model virtually unlimited examples of target distributions.

Can DiScoFormer handle distributions it has not seen before?

Yes. The model stays accurate on mixtures with more modes than it saw during training and on non-Gaussian shapes like the Laplace and Student-t distributions. It can also adapt itself to out-of-distribution inputs at inference time using a label-free consistency loss.