Special Architecture in AFM3 20B: instruction-following pruning reduces active parameters via MoE sparsity

https://openreview.net/forum?id=juARG7yu4P This is a model designed to activate ~20% of active MLP layers. It is also an MoE so it has some sparsity built-in. It's trained from scratch to use the same experts per...