ggml/llama.cpp adds --spec-draft-n-ubatch option to optimize VRAM usage
Read the original at www.reddit.com→a new option to optimize your VRAM submitted by /u/jacek2023 [link] [comments]
Original headline: "common : add --spec-draft-n-ubatch by ggerganov · Pull Request #30459 · ggml-org/llama.cpp"
Coverage timeline
- Oct 14, 11:08 UTC r/LocalLLaMA lead source common : add --spec-draft-n-ubatch by ggerganov · Pull Request #30459 · ggml-org/llama.cpp