SonicBoom achieves 52 tok/s decode on a laptop RTX 3050 (llama.cpp 45) with the same quantized kernels; the difference is a graph compiler, with no Python involved.
Read the original at www.reddit.com→I’ve been building SonicBoom, a native tensor runtime written in C++/CUDA with a C ABI and no Python anywhere in the stack. I just tagged v0.0.1, and the first target is local LLM inference on consumer GPUs. Numbers...
Original headline: "SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python."
Coverage timeline
- Oct 13, 09:46 UTC r/LocalLLaMA lead source SonicBoom: 52 tok/s decode on a laptop RTX 3050 (llama.cpp: 45) with the same quantized kernels, the difference is a graph compiler. No Python.