Cover Image

Tracing How Annotators Think: Augmenting Preference Judgments with Reading Processes

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	de Langis, Karin, Walker, William, Le, Khanh Chi, Kang, Dongyeop
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation
by: de Langis, Karin, et al.
Published: (2024)

How LLMs Comprehend Temporal Meaning in Narratives: A Case Study in Cognitive Evaluation of LLMs
by: de Langis, Karin, et al.
Published: (2025)

Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLMs
by: de Langis, Karin, et al.
Published: (2025)

Effects of Varying LLM Access on Essay Writing Behavior
by: Christenson, Julia, et al.
Published: (2026)

Mary, the Cheeseburger-Eating Vegetarian: Do LLMs Recognize Incoherence in Narratives?
by: de Langis, Karin, et al.
Published: (2025)

ScholaWrite: A Dataset of End-to-End Scholarly Writing Process
by: Le, Khanh Chi, et al.
Published: (2025)

LawFlow: Collecting and Simulating Lawyers' Thought Processes on Business Formation Case Studies
by: Das, Debarati, et al.
Published: (2025)

Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
by: Kim, Dongyoung, et al.
Published: (2024)

Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Models
by: Kim, Zae Myung, et al.
Published: (2025)

How Do Answer Tokens Read Reasoning Traces? Self-Reading Patterns in Thinking LLMs for Quantitative Reasoning
by: Chen, Haoyang, et al.
Published: (2026)

SelectLLM: Can LLMs Select Important Instructions to Annotate?
by: Parkar, Ritik Sachin, et al.
Published: (2024)

How Far Can We Extract Diverse Perspectives from Large Language Models?
by: Hayati, Shirley Anugrah, et al.
Published: (2023)

How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluation
by: Lee, Wilson Y.
Published: (2026)

From Blind Guess to Informed Judgment: Teaching LLMs to Evaluate Materials by Building Knowledge-Augmented Preference Signals
by: Yu, Yeyong, et al.
Published: (2026)

Beyond the Surface: Measuring Self-Preference in LLM Judgments
by: Chen, Zhi-Yuan, et al.
Published: (2025)

Structure Liberates: How Constrained Sensemaking Produces More Novel Research Output
by: Mooney, James, et al.
Published: (2026)

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL
by: Zhai, Skylar, et al.
Published: (2026)

Permutative Preference Alignment from Listwise Ranking of Human Judgments
by: Zhao, Yang, et al.
Published: (2024)

ASP2LJ : An Adversarial Self-Play Laywer Augmented Legal Judgment Framework
by: Chang, Ao, et al.
Published: (2025)

REPA: Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilities
by: Pugachev, Alexander, et al.
Published: (2025)

Mitigating Judgment Preference Bias in Large Language Models through Group-Based Polling
by: Liu, Shuliang, et al.
Published: (2025)

BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence
by: Sheng, Zhecheng, et al.
Published: (2023)

Shallow Synthesis of Knowledge in GPT-Generated Texts: A Case Study in Automatic Related Work Composition
by: Martin-Boyle, Anna, et al.
Published: (2024)

An Annotated Reading of 'The Singer of Tales' in the LLM Era
by: Varshney, Kush R.
Published: (2025)

MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning
by: Huang, Kun, et al.
Published: (2025)

Good/Evil Reputation Judgment of Celebrities by LLMs via Retrieval Augmented Generation
by: Tsuchida, Rikuto, et al.
Published: (2025)

AnnoCaseLaw: A Richly-Annotated Dataset For Benchmarking Explainable Legal Judgment Prediction
by: Sesodia, Magnus, et al.
Published: (2025)

Which Modality should I use -- Text, Motif, or Image? : Understanding Graphs with Large Language Models
by: Das, Debarati, et al.
Published: (2023)

II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering
by: Kil, Jihyung, et al.
Published: (2024)

A Multilingual Human Annotated Corpus of Original and Easy-to-Read Texts to Support Access to Democratic Participatory Processes
by: Bott, Stefan, et al.
Published: (2026)

Do LLMs Recognize Your Latent Preferences? A Benchmark for Latent Information Discovery in Personalized Interaction
by: Tsaknakis, Ioannis, et al.
Published: (2025)

How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality
by: Tu, Minzhu, et al.
Published: (2026)

Reasoning Beyond Literal: Cross-style Multimodal Reasoning for Figurative Language Understanding
by: Cheshmi, Seyyed Saeid, et al.
Published: (2026)

Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation
by: Yang, Ruixin, et al.
Published: (2024)

SEMDR: A Semantic-Aware Dual Encoder Model for Legal Judgment Prediction with Legal Clue Tracing
by: Liu, Pengjie, et al.
Published: (2024)

Human-AI Collaborative Taxonomy Construction: A Case Study in Profession-Specific Writing Assistants
by: Lee, Minhwa, et al.
Published: (2024)

A2P-Vis: an Analyzer-to-Presenter Agentic Pipeline for Visual Insights Generation and Reporting
by: Gan, Shuyu, et al.
Published: (2025)

Enhancing the Preference Extractor in Multi-turn Dialogues: From Annotating Disasters to Accurate Preference Extraction
by: Wang, Cheng, et al.
Published: (2025)

Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning
by: Yang, Sen, et al.
Published: (2024)

PORT: Preference Optimization on Reasoning Traces
by: Lahlou, Salem, et al.
Published: (2024)