Investigating quantization of k and v caches; user reports increased speed over longer tasks but seeks others’ experiences
Read the original at old.reddit.com→i so far have not felt a difference other than that you get more speed the longer your task goes for but i am curious for your experiences submitted by /u/False-Advantage-4984 [link] [comments]
Original headline: "Do you currently quantazise your k and v caches?"
Coverage timeline
- Aug 16, 11:44 UTC r/LocalLLaMA lead source Do you currently quantazise your k and v caches?