Scaling Laws for Neural Language Models
Library
Three power laws. One unified prediction surface.
Language model performance does not depend on how you build it.
It depends on how much you scale it.
A reading of Scaling Laws for Neural Language Models
by Kaplan, McCandlish, Henighan, Brown et al.
Model size N. Dataset size D. Compute budget C.
Each predicts loss. Each spans six orders of magnitude.
— where optimal training lives
For any fixed compute budget, there is an exactly optimal model size. It is almost always much larger than practitioners currently train. The paper shows: train a bigger model, stop early, use fewer tokens. Larger models reach the same loss with fewer examples. This is not a compromise — it is the frontier.
The paper demolishes two myths. First: that you need to tune architecture — depth, width, attention heads, layer norm placement. The data says no. Scale dominates. Second: that overfitting is the constraint. The paper shows overfitting is governed by the ratio N^0.74/D — meaning dataset size needs to grow only sub-linearly as models grow. A 10x larger model needs only 5x more data. This inverts the training playbook. Stop training small models to convergence. Train large models and stop early. You will reach the same loss faster and cheaper.
The compute-efficient frontier almost always points toward larger models trained briefly.
We read every message. No algorithm, no auto-reply — just Sophie and the LOOMUS team, with coffee, sometime this week.
Your note is in the inbox.
We'll write back when the kettle's on.
Saved with your account — part of a study that grows every time you distill. Nothing here expires, and nothing is lost.
Every LOOMUS reader may claim a planet — once, for life. It’s a slow choice, made when you’re ready.
Enter The Choosing →or simply look around your space →
No hurry; the sky holds your seat.