A record of increasing MFU from 21% to 50% during pre-training of a 21B MoE on a single GH200 chip (includes code and logs)
Read the original at www.reddit.com→I am an independent researcher. Due to budget constraints, I shifted my 21B MoE pre-training—which I had originally planned to run on 64 H100s—to a single NVIDIA GH200 ($2.29 per hour), and I have compiled the...
Original headline: "A Record of the Process of Increasing MFU from 21% to 50% While Pre-training 21B MoE Using a Single GH200 Chip (Includes Code and Logs)"
Coverage timeline
- Oct 12, 12:52 UTC r/LocalLLaMA lead source A Record of the Process of Increasing MFU from 21% to 50% While Pre-training 21B MoE Using a Single GH200 Chip (Includes Code and Logs)