Disabling MTP can improve token throughput when running very large models on systems with limited RAM, according to testing with gyrostrata and strata on a 3060 and 32GB RAM.
Read the original at www.reddit.com→Little reminder, with all my testing I found out that mtp can decrease performance when using very big model, since many are starting to use strata or similar and I see many app activating it as default try and...
Original headline: "If you are using big model compared to ram try and disable mtp"
Coverage timeline
- Nov 2, 09:06 UTC r/LocalLLaMA lead source If you are using big model compared to ram try and disable mtp