CPU decode speed depends on active parameters per token, not total; aiming to run 10B at 100 tok/s on a mid-tier PC without GPU
Read the original at old.reddit.com→On the CPU, batch 1 is memory bandwidth bound. But if token/s = bandwidth / (bytes_per_weight * active_weights_per_token) the total number of parameters doesnt slow down the generation speed. So building the...
Original headline: "The idea: on a CPU the decode speed depends on the active params per token, not the total. My objective is trying to run a 10B at 100tok/s on a mid level PC (No GPU)."
Coverage timeline
- Jul 29, 13:06 UTC r/LocalLLaMA lead source The idea: on a CPU the decode speed depends on the active params per token, not the total. My objective is trying to run a 10B at 100tok/s on a mid level PC (No GPU).