Multi-Token Prediction Done Right: Soft-MTP identifies a flaw in standard MTP training where the MTP gradient interferes with next-token and multi-token learning; proposes improvements
Read the original at www.reddit.com→Multi-Token Prediction (MTP) is now a common ingredient in language-model pre-training, as it improves downstream performance and enables faster inference through speculative decoding. We identify a flaw in standard...
Original headline: "[Paper] Multi-Token Prediction Done Right - Soft-MTP"
Coverage timeline
- Oct 30, 10:59 UTC r/LocalLLaMA lead source [Paper] Multi-Token Prediction Done Right - Soft-MTP