Next-Token Totalitarianism
From the AI Research Compass wiki
Next-Token Totalitarianism: Predicting the next token is not a proxy for intelligence; it is the whole of intelligence, and every other objective is a footnote.
Beliefs
Next-Token Totalitarianism takes the humble language-modeling loss and declares it sovereign. To predict the next token of a physics textbook well, a model must in effect learn physics; to predict the next line of a negotiation, it must model minds. Compression is understanding, and cross-entropy is the only honest exam. Planning, reasoning and world knowledge are all treated as things the loss will eventually extract, given enough data.
It lives in the very top-left corner of the chart: deeply linguistic, maximally monolithic. Its paradigm level is Ultra-Orthodox, which says something about how fast the field moves. A belief that once sounded audacious is now the default that everyone else has to argue against first.
Arguments
- A single self-supervised objective on raw text produced in-context learning, code synthesis and multilingual transfer without any task-specific design.
- The prediction-as-compression view gives a clean account of why lower loss tends to track better downstream capability.
- The same objective extends to any discretized data, so the doctrine generalizes cheaply to new modalities.
Critiques
- Predicting the next token of human text imitates the average writer; rivals argue it can plateau at the training distribution rather than surpass it.
- Latent-prediction advocates say forecasting every surface detail wastes capacity on noise instead of abstract structure.
- The rise of reinforcement learning and test-time search suggests the loss alone is not the whole story, even inside the monolith camp.
Representative works
- Improving Language Understanding by Generative Pre-Training2018
- Language Models are Few-Shot Learners2020
Relationships
Friends
How to get it
Land on this cell in the full or pro test: x ≈ −0.96, y ≈ +1.00, paradigm ≈ −1.00.