Two ~300B MoE models fit on one 128 GB mini PC each (GLM-5.3-Flash and MiMo-V2.6-Flash) running on EXL3/ROCm: prefill speeds around 580 tok/s for GLM-5.3-Flash and up to 44 tok/s decode for MiMo-V2.6-Flash.
Read the original at www.reddit.com→We built an engine, Kyojin, on top of ExLlamaV3 for Strix Halo (gfx1151, ROCm), and packed two 300B-class MoE models so each fits one 128 GB machine. First release, all measured on Ryzen AI Max+ 395. Model...
Original headline: "Two ~300B MoE models, each on ONE 128 GB mini PC (AMD Strix Halo): GLM-5.3-Flash at ~580 tok/s prefill, MiMo-V2.6-Flash up to 44 tok/s decode. EXL3 weights + open ROCm engine"