PyTorch "dynamic notch attention filter" targets hallucination via PEQ math

The proposal starts from an analogy: multi-head attention and positional embeddings such as RoPE act like a spectrum decomposer, splitting work across different frequency characteristics and feature spaces. What standard attention lacks, the author argues, is an explicit control layer comparable to the pinpoint attenuation (notch filtering) and dynamic Q adjustment used in audio signal processing.

The mathematical model maps that idea onto the attention matrix. In a peaking/notch filter, the frequency response is governed by a center frequency and a Q value that sets bandwidth, and the same shape applies here: a Gaussian notch mask is generated dynamically over the similarity between keys and a target noise embedding, with high Q producing a narrow sharp cut and low Q damping a wider span. A separate notch gain acts as the penalty coefficient that pushes down scores in the region flagged as noise.

The implementation sketch ties the two parts together. It computes ordinary attention scores, measures the distance between averaged keys and the target embeddings, converts the minimum distance into a Gaussian notch mask, and subtracts the scaled penalty before the softmax. The write-up concludes that importing parametric EQ and notch-filter mathematics into attention could embed pinpoint removal of unwanted noise components, which it describes as an effective control method for free-form text where probabilistic fluctuation and hallucination easily creep in.