Eval Proceduralism

From the AI Research Compass wiki

Eval Proceduralism: Progress exists only when measured by careful, reproducible evaluation procedures, so building good evals is the core research work.

Beliefs

Eval Proceduralism holds that a claim without a well-designed evaluation is an anecdote. Due process matters: held-out splits, contamination checks, multiple seeds, confidence intervals, clear metrics and baselines run fairly. Benchmarks are the field’s courts, and like courts they need procedure to be trusted. When a capability is hard to measure, the proceduralist’s answer is not to give up but to build a better test, because whatever cannot be measured will be argued about forever.

It sits on the Language side, left of centre and just above the middle, since most contested benchmarks today are language tasks and since evaluation applies equally to monoliths and harnesses. Its steady shade reflects a measured stance close to the mainstream: evaluation is how every paradigm proves itself. It is a cousin of leaderboard centrism, but stricter; the proceduralist cares less about who is first and more about whether the ranking means anything.

Arguments

  • Shared benchmarks have repeatedly coordinated progress, giving the field common targets and making methods comparable.
  • Contamination, prompt sensitivity and seed variance can swing results substantially, so rigorous procedure is what separates real gains from noise.
  • Building an eval forces a precise definition of the capability, which often clarifies the research question itself.

Critiques

  • Benchmarks saturate and get overfit, so procedure can produce precise measurements of the wrong thing.
  • Open-ended and agentic capabilities resist static evaluation, and insisting on metrics can bias research toward what is easy to score.
  • Fast-moving camps argue that heavy evaluation process slows iteration and that users vote with deployment.

Representative works

Relationships

How to get it

Land on this cell in the full or pro test: x ≈ −0.49, y ≈ −0.63, paradigm ≈ −0.33.

A first draft. Corrections and better arguments are welcome through a pull request. Edit this entry on GitHub Edit its data