Any-to-Any Imperialism

From the AI Research Compass wiki

Any-to-Any Imperialism: One model should take any modality as input and produce any modality as output, annexing every specialist system along the way.

Beliefs

Any-to-Any Imperialism looks at the zoo of specialist models (a captioner here, a text-to-speech system there, an image generator somewhere else) and sees provinces waiting to be unified. Its goal is a single network that maps any combination of text, image, audio and video to any other combination. Every specialist that survives is, in this view, just a capability the empire has not yet absorbed.

It sits at the very top of the chart, a little right of centre. The Monolith position reflects its expansionism: capabilities should be added by training, not by bolting on another model. Its horizontal position is only slightly right of centre, because it is genuinely modality-agnostic, with no special loyalty to language or to perception. Its paradigm score is neutral: few production systems have fully achieved it, but the attempt extends today’s scaled Transformer recipe rather than abandoning it.

Arguments

  • Cross-modal transfer is strongest when modalities share weights; a model that both hears and speaks can learn prosody from both directions.
  • Removing hand-offs between specialist models removes compounding errors and latency at every boundary.
  • Research systems spanning vision, language, audio and action in one autoregressive model show the approach can work at meaningful scale.

Critiques

  • Jack-of-all-modalities models often trail dedicated specialists on any single task.
  • Training mixtures across many modalities are hard to balance, and one modality can quietly dominate or degrade the others.
  • Harness advocates point out that routing to specialist models is cheaper, easier to update and often good enough.

Representative works

Relationships

How to get it

Land on this cell in the full or pro test: x ≈ +0.62, y ≈ +0.91, paradigm ≈ −0.01.

A first draft. Corrections and better arguments are welcome through a pull request. Edit this entry on GitHub Edit its data