Synthetic-Data Planism
From the AI Research Compass wiki
Synthetic-Data Planism: When natural data runs short, models should generate their own training data under planned, verified pipelines.
Beliefs
Synthetic-Data Planism answers the data wall with a printing press. If the supply of high-quality human text is finite, then models must produce more of it: instructions written by models, textbook-style explanations generated on demand, reasoning traces filtered by verifiers, captions rewritten by stronger captioners. Like a planned economy, the key is not raw production but quotas and quality control. Decide what the next model needs, generate it, filter it, and train.
It sits in the upper half, close to the Language edge. Its horizontal position leans hard toward Language because its main product is model-written text: instructions, explanations and reasoning traces. The Monolith lean reflects that the end product is still one model trained on a planned corpus. Its paradigm level is Mainstream: synthetic data has moved from a curiosity to a core part of many training pipelines.
Arguments
- Bootstrapping instruction data from a model’s own generations can substantially improve instruction following at low cost.
- Small models trained on carefully generated textbook-quality data have performed far above what their size would predict on coding benchmarks.
- Self-generated reasoning traces, filtered for correct answers, can iteratively improve a model’s own reasoning.
Critiques
- Recursively training on model outputs risks degrading diversity and amplifying errors, a failure often called model collapse.
- Synthetic data can only be as good as the generator and the verifier; it rarely adds knowledge the system did not already have.
- Benchmark gains can partly reflect contamination or stylistic overfitting rather than genuine capability.
Representative works
- Self-Instruct: Aligning Language Models with Self-Generated Instructions2022
- STaR: Bootstrapping Reasoning With Reasoning2022
- Textbooks Are All You Need2023
Relationships
Friends
How to get it
Land on this cell in the full or pro test: x ≈ −0.84, y ≈ +0.60, paradigm ≈ −0.62.