Multimodal Linguism
From the AI Research Compass wiki
Multimodal Linguism: Holds that a model may perceive through many senses, but its thinking should happen in language.
Beliefs
Multimodal Linguism follows the naming rule exactly: language is the substance, multimodality the means. Eyes and ears are welcome, but they are input devices. Whatever comes in, whether an image, a chart or an audio clip, is translated into the model’s linguistic workspace, and that is where reasoning, planning and explanation take place. Language is the most general representational system humans have, so it is the natural place for a mind to think, even when what it thinks about is visual.
It sits at the far Language edge, and in the upper half of the vertical axis: one model, with perception folded in. Its paradigm level is Orthodox, since it describes the architecture of many current vision-language assistants, where image tokens flow into a language model that does the talking and the thinking.
Arguments
- Vision-language models built this way handle captioning, visual question answering and document reading well, while inheriting the LM’s broad reasoning and knowledge.
- Language offers an interpretable chain of thought; reasoning in words can be inspected, critiqued and corrected in a way that hidden visual activations cannot.
- Composing perception models through a language interface has shown strong zero-shot multimodal reasoning with little or no joint training.
Critiques
- Its mirror image, Linguistic Multimodalism, argues the goal should be multimodal outputs too, not just language about images.
- Vision-first camps say forcing perception through words loses spatial and geometric detail that language describes poorly.
- Language bottlenecks can make models confidently describe things that are not in the image at all.
Representative works
- Flamingo: a Visual Language Model for Few-Shot Learning2022
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language2022
Relationships
Friends
Frenemies
How to get it
Land on this cell in the full or pro test: x ≈ −1.00, y ≈ +0.17, paradigm ≈ −0.70.