Researchers show that training data significantly impacts draft model performance in speculative decoding, with task-specific datasets like MathInstruct outperforming generic training on specialized benchmarks.

Study evaluates lightweight HASS and EAGLE-2 draft models trained on MathInstruct, ShareGPT, and mixed-data variants to understand how training distribution affects speculative decoding quality

MathInstruct-trained drafts excel on reasoning benchmarks (GSM8K, MATH-500, SVAMP), while ShareGPT-trained drafts perform best on general MT-Bench evaluations

Mixed-data training improves robustness across different tasks, but larger data mixtures don't uniformly improve performance across all decoding temperatures

Research explores combining specialized drafters at inference time as an alternative to traditional checkpoint averaging for better cross-task performance