Apex-2 training scale and bottlenecks: GPU/VRAM limits and lessons from the build process

Hi everyone, thank you so much for all the interest in my model. It's more than I expected. Here is a short summary of the trial and error I went through while building Apex-2. 1. GPUs were always the bottleneck I...