Running 35B MoE models on a Tesla K80 (2014) at about 21 tokens per second in plain C

I've spent about a year on an inference engine that doesn't need modern hardware. It's called Pantheon (LLM engine) on top of Foundry (ML library), both in C with no outside libraries. The CUDA backend targets Kepler...