Distributional Semanticism

From the AI Research Compass wiki

Distributional Semanticism: A word's meaning is the pattern of contexts it appears in, so meaning can be learned from co-occurrence statistics alone.

Beliefs

Distributional Semanticism rests on one old slogan: you shall know a word by the company it keeps. Words that appear in similar contexts have similar meanings, so a model that captures the statistics of contexts captures meaning, no dictionary, logic or perception required. Word embeddings made this concrete as geometry, with analogies as vector offsets, and contextual language models extended it from words to whole passages. In this view, next-token prediction is not a trick that happens to work; it is distributional semantics carried to its conclusion.

It sits near the Language edge, a little above the middle. It is a theory of meaning rather than an architecture, so it is largely agnostic between monoliths and systems, but it firmly locates meaning in text. Its orthodox shade marks an established view at the heart of the mainstream: the hypothesis is decades old and has quietly become the foundation of modern NLP, which makes its defenders more confident than combative.

Arguments

  • Embeddings learned purely from co-occurrence recover synonymy, analogy and semantic clusters that align well with human similarity judgments.
  • Every large language model is in effect a distributional learner, and their broad semantic competence is strong evidence for the hypothesis.
  • The approach needs only raw text, so it scales to any language or domain with a corpus, without annotation or ontology.

Critiques

  • Grounding camps argue that co-occurrence can tell you that ‘red’ is near ‘crimson’ but not what red looks like, so meaning is incomplete.
  • Formal semanticists note that distributional similarity blurs crucial logical distinctions such as antonyms appearing in near-identical contexts.
  • Statistical meaning inherits the biases and gaps of the corpus, so what is learned reflects usage, not truth.

Representative works

Relationships

How to get it

Land on this cell in the full or pro test: x ≈ −0.81, y ≈ +0.27, paradigm ≈ −0.87.

Categories: Linguism · Paradigm: Orthodox

A first draft. Corrections and better arguments are welcome through a pull request. Edit this entry on GitHub Edit its data