Character.ai shares early techniques for faster, more efficient large-scale transformer pretraining; Noam Shazeer co-led the effort

Before Character.ai shifted its focus toward building on open-source model foundations, the company’s early pretraining team explored a range of techniques to make large-scale transformer training faster and more...