Running decision models: picking quantization and VRAM usage
Read the original at www.reddit.com→Hi, I see that vllm released d3 (a family of decision models) and llama.cpp already has support for them. I want to try them out. Are decision models sensitive to quantization? I know that for autocomplete (FIM)...
Original headline: "Running decision models: Picking quant, VRAM usage"
Coverage timeline
- Oct 13, 06:28 UTC r/LocalLLaMA lead source Running decision models: Picking quant, VRAM usage