FreqFormer introduces frequency-aware heterogeneous attention for long-sequence video diffusion transformers, splitting tokens into spectral bands with different operators to reduce attention costs.

FreqFormer applies different attention methods to different frequency bands in video features: dense global attention on compressed low-frequency content, structured block-sparse attention on mid frequencies, and sliding-window local attention on high frequencies.

A lightweight spectral routing network allocates computational heads across bands using layer statistics and the diffusion timestep, shifting compute toward global structure early in denoising and detail later. Cross-band summary tokens provide residual exchange between bands.

In simulations from 64K to 1M tokens, FreqFormer substantially reduces estimated attention FLOPs and KV-related memory traffic versus dense attention while maintaining a hardware-friendly pattern, supported by a fused GPU execution plan that co-schedules dense, sparse, and local branches.