A survey of confidence estimation and calibration in large language models

J Geng, F Cai, Y Wang, H Koeppl, P Nakov… - arxiv preprint arxiv …, 2023 - arxiv.org
Large language models (LLMs) have demonstrated remarkable capabilities across a wide
range of tasks in various domains. Despite their impressive performance, they can be …

Uncertainty in natural language processing: Sources, quantification, and applications

M Hu, Z Zhang, S Zhao, M Huang, B Wu - arxiv preprint arxiv:2306.04459, 2023 - arxiv.org
As a main field of artificial intelligence, natural language processing (NLP) has achieved
remarkable success via deep neural networks. Plenty of NLP tasks have been addressed in …

Robots that ask for help: Uncertainty alignment for large language model planners

AZ Ren, A Dixit, A Bodrova, S Singh, S Tu… - arxiv preprint arxiv …, 2023 - arxiv.org
Large language models (LLMs) exhibit a wide range of promising capabilities--from step-by-
step planning to commonsense reasoning--that may provide utility for robots, but remain …

[HTML][HTML] Generative AI in EU law: Liability, privacy, intellectual property, and cybersecurity

C Novelli, F Casolari, P Hacker, G Spedicato… - Computer Law & Security …, 2024 - Elsevier
The complexity and emergent autonomy of Generative AI systems introduce challenges in
predictability and legal compliance. This paper analyses some of the legal and regulatory …

Evaluating language models for mathematics through interactions

KM Collins, AQ Jiang, S Frieder, L Wong… - Proceedings of the …, 2024 - pnas.org
There is much excitement about the opportunity to harness the power of large language
models (LLMs) when building problem-solving assistants. However, the standard …

Bayesian low-rank adaptation for large language models

AX Yang, M Robeyns, X Wang, L Aitchison - arxiv preprint arxiv …, 2023 - arxiv.org
Low-rank adaptation (LoRA) has emerged as a new paradigm for cost-efficient fine-tuning of
large language models (LLMs). However, fine-tuned LLMs often become overconfident …

Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models

A Amayuelas, K Wong, L Pan, W Chen… - arxiv preprint arxiv …, 2023 - arxiv.org
This paper investigates the capabilities of Large Language Models (LLMs) in the context of
understanding their knowledge and uncertainty over questions. Specifically, we focus on …

Decomposing uncertainty for large language models through input clarification ensembling

B Hou, Y Liu, K Qian, J Andreas, S Chang… - arxiv preprint arxiv …, 2023 - arxiv.org
Uncertainty decomposition refers to the task of decomposing the total uncertainty of a
predictive model into aleatoric (data) uncertainty, resulting from inherent randomness in the …

Shifting attention to relevance: Towards the uncertainty estimation of large language models

J Duan, H Cheng, S Wang, A Zavalny, C Wang, R Xu… - 2023 - openreview.net
While Large Language Models (LLMs) have demonstrated remarkable potential in natural
language generation and instruction following, a persistent challenge lies in their …

Luq: Long-text uncertainty quantification for llms

C Zhang, F Liu, M Basaldella, N Collier - arxiv preprint arxiv:2403.20279, 2024 - arxiv.org
Large Language Models (LLMs) have demonstrated remarkable capability in a variety of
NLP tasks. However, LLMs are also prone to generate nonfactual content. Uncertainty …