Kaplan, Chinchilla, and broken laws
That paper shaped how labs designed pre-training experiments for the next two years, and the eventual "Chinchilla" effort grew out of trying to reproduce and extend its recommendations. It also turned out…
Double descent
Nakkiran et al. [2] made the picture concrete by showing that the W-shape appears in three different axes: model size…