Mica v0.5, a 4B model on a single GPU, achieves 19.8 ms per decision and up to 25x faster than Jev across benchmarks.
Read the original at www.reddit.com→I'm sharing Mica v0.5, the follow-up to Mica v0.1. It's 4B and runs on one consumer GPU. On speed, it takes 19.8 ms per decision at 256 tokens and about 1.2 s at 32k on a single RTX 6000 Ada. That's up to 3.8x faster...
Original headline: "Mica v0.5: The smartest 4B decision model is also the fastest one. 20 ms per answer, 25x faster than Jev"
Coverage timeline
- Oct 21, 19:00 UTC r/LocalLLaMA lead source Mica v0.5: The smartest 4B decision model is also the fastest one. 20 ms per answer, 25x faster than Jev