Symbols and mental programs: a hypothesis about human singularity
Natural language is often seen as the single factor that explains the cognitive singularity of
the human species. Instead, we propose that humans possess multiple internal languages …
the human species. Instead, we propose that humans possess multiple internal languages …
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models
Language models demonstrate both quantitative improvement and new qualitative
capabilities with increasing scale. Despite their potentially transformative impact, these new …
capabilities with increasing scale. Despite their potentially transformative impact, these new …
Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models
Abstract Language models learn a great quantity of factual information during pretraining,
and recent work localizes this information to specific model weights like mid-layer MLP …
and recent work localizes this information to specific model weights like mid-layer MLP …
Masked language modeling and the distributional hypothesis: Order word matters pre-training for little
A possible explanation for the impressive performance of masked language model (MLM)
pre-training is that such models have learned to represent the syntactic structures prevalent …
pre-training is that such models have learned to represent the syntactic structures prevalent …
A taxonomy and review of generalization research in NLP
The ability to generalize well is one of the primary desiderata for models of natural language
processing (NLP), but what 'good generalization'entails and how it should be evaluated is …
processing (NLP), but what 'good generalization'entails and how it should be evaluated is …
Is Ockham's razor losing its edge? New perspectives on the principle of model parsimony
The preference for simple explanations, known as the parsimony principle, has long guided
the development of scientific theories, hypotheses, and models. Yet recent years have seen …
the development of scientific theories, hypotheses, and models. Yet recent years have seen …
State-of-the-art generalisation research in NLP: a taxonomy and review
The ability to generalise well is one of the primary desiderata of natural language
processing (NLP). Yet, what'good generalisation'entails and how it should be evaluated is …
processing (NLP). Yet, what'good generalisation'entails and how it should be evaluated is …
Causal analysis of syntactic agreement mechanisms in neural language models
Targeted syntactic evaluations have demonstrated the ability of language models to perform
subject-verb agreement given difficult contexts. To elucidate the mechanisms by which the …
subject-verb agreement given difficult contexts. To elucidate the mechanisms by which the …
On the proper role of linguistically oriented deep net analysis in linguistic theorising
M Baroni - Algebraic structures in natural language, 2022 - taylorfrancis.com
A lively research field has recently emerged that uses experimental methods to probe the
linguistic behaviour of modern deep networks. While work in this tradition often reports …
linguistic behaviour of modern deep networks. While work in this tradition often reports …
Language models as models of language
R Millière - arxiv preprint arxiv:2408.07144, 2024 - arxiv.org
This chapter critically examines the potential contributions of modern language models to
theoretical linguistics. Despite their focus on engineering goals, these models' ability to …
theoretical linguistics. Despite their focus on engineering goals, these models' ability to …