Muse Glimmer 30B with 512k context and unusual long-context architecture; author experiments with extending context to 512k and notes RoPE presence only in 2048-token SWA layers
Read the original at old.reddit.com→My fun weekend project was to try to make the new Muse Glimmer 30B work with a longer context, deciding to go for 512k first. I had expected the usual YaRN shenanigans and maybe a LoRA. I couldn't have been wrong...
Original headline: "Muse Glimmer 30B with 512k context"
Coverage timeline
- Aug 17, 09:50 UTC r/LocalLLaMA lead source Muse Glimmer 30B with 512k context