SRA framework reframes cross-tokenizer knowledge distillation by aligning spans rather than tokens, using a physics-based approach with Center of Mass representations.

Researchers introduced SRA (Span Representation Alignment), a method for knowledge distillation between large language models and smaller student models that use different tokenizers, modeling each span as a cluster of particles represented by its Center of Mass—an attention-weighted average.

SRA shifts the fundamental unit of alignment from individual tokens to tokenizer-agnostic spans, employs a geometric regularizer to preserve structural integrity of the representation space, and introduces aligned span logit distillation to enhance knowledge transfer across models.

In cross-architecture distillation experiments, SRA consistently outperformed state-of-the-art cross-tokenizer knowledge distillation baselines, according to the research.