2-bit copy in VRAM + exact residual in system RAM as the verifier: speculative decoding for models that don't fit your GPU
Read the original at www.reddit.com→Hi Guys!, Recently I was making a research and development, but I do not have time to go further with this idea (kids, work, etc.). But maybe it will be interesting for someone. I do not think it would be something...
Coverage timeline
- Oct 22, 01:18 UTC r/LocalLLaMA lead source 2-bit copy in VRAM + exact residual in system RAM as the verifier: speculative decoding for models that don't fit your GPU