Psychometric predictive power of large language models

T Kuribayashi, Y Oseki, T Baldwin - arxiv preprint arxiv:2311.07484, 2023 - arxiv.org
Instruction tuning aligns the response of large language models (LLMs) with human
preferences. Despite such efforts in human--LLM alignment, we find that instruction tuning …

Holmes ⌕ A Benchmark to Assess the Linguistic Competence of Language Models

A Waldis, Y Perlitz, L Choshen, Y Hou… - Transactions of the …, 2024 - direct.mit.edu
We introduce Holmes, a new benchmark designed to assess language models'(LMs')
linguistic competence—their unconscious understanding of linguistic phenomena …

Can Large Language Models Interpret Noun-Noun Compounds? A Linguistically-Motivated Study on Lexicalized and Novel Compounds

G Rambelli, E Chersoni, C Collacciani… - Proceedings of the …, 2024 - aclanthology.org
Noun-noun compounds interpretation is the task where a model is given one of such
constructions, and it is asked to provide a paraphrase, making the semantic relation …

Brain-like language processing via a shallow untrained multihead attention network

B AlKhamissi, G Tuckute, A Bosselut… - arxiv preprint arxiv …, 2024 - arxiv.org
Large Language Models (LLMs) have been shown to be effective models of the human
language system, with some models predicting most explainable variance of brain activity in …

Large Language Models Are Human-Like Internally

T Kuribayashi, Y Oseki, SB Taieb, K Inui… - arxiv preprint arxiv …, 2025 - arxiv.org
Recent cognitive modeling studies have reported that larger language models (LMs) exhibit
a poorer fit to human reading behavior, leading to claims of their cognitive implausibility. In …

Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models

C Kauf, E Chersoni, A Lenci, E Fedorenko… - arxiv preprint arxiv …, 2024 - arxiv.org
Semantic plausibility (eg knowing that" the actor won the award" is more likely than" the
actor won the battle") serves as an effective proxy for general world knowledge. Language …

From Words to Worlds: Compositionality for Cognitive Architectures

R Dhar, A Søgaard - arxiv preprint arxiv:2407.13419, 2024 - arxiv.org
Large language models (LLMs) are very performant connectionist systems, but do they
exhibit more compositionality? More importantly, is that part of why they perform so well? We …

Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference

S Wang, D Wang, C Zhou, Z Li, Z Fan, X Huang… - arxiv preprint arxiv …, 2024 - arxiv.org
Large Vision-Language Models (LVLMs) typically learn visual capacity through visual
instruction tuning, involving updates to both a projector and their LLM backbones. Drawing …

The potential--and the pitfalls--of using pre-trained language models as cognitive science theories

RS Shah, S Varma - arxiv preprint arxiv:2501.12651, 2025 - arxiv.org
Many studies have evaluated the cognitive alignment of Pre-trained Language Models
(PLMs), ie, their correspondence to adult performance across a range of cognitive domains …

On Representational Dissociation of Language and Arithmetic in Large Language Models

R Kisako, T Kuribayashi, R Sasano - arxiv preprint arxiv:2502.11932, 2025 - arxiv.org
The association between language and (non-linguistic) thinking ability in humans has long
been debated, and recently, neuroscientific evidence of brain activity patterns has been …