Proj. CMI Paper Reading: R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents

Abstract

Task: building uncertainty-aware suggestions based on a decision-theoretic model of goal-conditional utility，推理LLM用户的未观测到的意图
方法：a decision-theoretic model of goal-conditioned utility, 使用生成式模型生成的random samples来做proxy， minimum Bayes-risk decoding, dual decomposition, decision diagrams.
实验：
数据集：3 developer-assistance tasks
效果：与per-token probability相比，获取more useful uncertainty estimates