Cover Image

Towards Understanding Bias in Synthetic Data for Evaluation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Rahmani, Hossein A., Ramineni, Varsha, Yilmaz, Emine, Craswell, Nick, Mitra, Bhaskar
Format:	Preprint
Published:	2025
Subjects:	Information Retrieval Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Synthetic Test Collections for Retrieval Evaluation
by: Rahmani, Hossein A., et al.
Published: (2024)

Understanding the Role of User Profile in the Personalization of Large Language Models
by: Wu, Bin, et al.
Published: (2024)

JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment
by: Rahmani, Hossein A., et al.
Published: (2024)

SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval
by: Rahmani, Hossein A., et al.
Published: (2024)

Overview of the TREC 2021 deep learning track
by: Craswell, Nick, et al.
Published: (2025)

Overview of the TREC 2023 deep learning track
by: Craswell, Nick, et al.
Published: (2025)

Overview of the TREC 2022 deep learning track
by: Craswell, Nick, et al.
Published: (2025)

Towards Group-aware Search Success
by: Wu, Haolun, et al.
Published: (2024)

Large language models can accurately predict searcher preferences
by: Thomas, Paul, et al.
Published: (2023)

Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024
by: Rahmani, Hossein A., et al.
Published: (2024)

Judging the Judges: A Collection of LLM-Generated Relevance Judgements
by: Rahmani, Hossein A., et al.
Published: (2025)

LLMJudge: LLMs for Relevance Judgments
by: Rahmani, Hossein A., et al.
Published: (2024)

Beyond Internal Data: Constructing Complete Datasets for Fairness Testing
by: Ramineni, Varsha, et al.
Published: (2025)

SocRipple: A Two-Stage Framework for Cold-Start Video Recommendations
by: Jaspal, Amit, et al.
Published: (2025)

Sociotechnical Implications of Generative Artificial Intelligence for Information Access
by: Mitra, Bhaskar, et al.
Published: (2024)

Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges
by: Thakur, Nandan, et al.
Published: (2025)

Image-Seeking Intent Prediction for Cross-Device Product Search
by: Hendriksen, Mariya, et al.
Published: (2025)

A Personalized Framework for Consumer and Producer Group Fairness Optimization in Recommender Systems
by: Rahmani, Hossein A., et al.
Published: (2024)

AgentSearchBench: A Benchmark for AI Agent Search in the Wild
by: Wu, Bin, et al.
Published: (2026)

Clarifying the Path to User Satisfaction: An Investigation into Clarification Usefulness
by: Rahmani, Hossein A., et al.
Published: (2024)

Query-Conditioned Graph Retrieval for Contextualized LLM Reasoning in Personalized Wearable Data
by: Lu, Zhenyu, et al.
Published: (2026)

Bridging Semantic Understanding and Popularity Bias with LLMs
by: Luo, Renqiang, et al.
Published: (2026)

InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem
by: Qiao, Shuofei, et al.
Published: (2026)

Interplay: Training Independent Simulators for Reference-Free Conversational Recommendation
by: Ramos, Jerome, et al.
Published: (2026)

Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
by: Pradeep, Ronak, et al.
Published: (2024)

Cluster-based Adaptive Retrieval: Dynamic Context Selection for RAG Applications
by: Xu, Yifan, et al.
Published: (2025)

Improving Vietnamese Legal Document Retrieval using Synthetic Data
by: Tien, Son Pham, et al.
Published: (2024)

Generating Diverse Synthetic Datasets for Evaluation of Real-life Recommender Systems
by: Malenšek, Miha, et al.
Published: (2024)

InPars+: Supercharging Synthetic Data Generation for Information Retrieval Systems
by: Krastev, Matey, et al.
Published: (2025)

Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation
by: Zhang, Benyu, et al.
Published: (2026)

Information Access of the Oppressed: Freirean Design for Emancipatory Information Access
by: Mitra, Bhaskar, et al.
Published: (2026)

Towards a Theoretical Understanding of Two-Stage Recommender Systems
by: Jaiswal, Amit Kumar
Published: (2024)

Bias vs Bias -- Dawn of Justice: A Fair Fight in Recommendation Systems
by: Kheya, Tahsin Alamgir, et al.
Published: (2025)

L2Rec: Towards Dual-View Understanding of LLMs for Personalized Recommendation
by: Pan, Pingjun, et al.
Published: (2026)

Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation
by: Balog, Krisztian, et al.
Published: (2025)

Toward Holistic Evaluation of Recommender Systems Powered by Generative Models
by: Deldjoo, Yashar, et al.
Published: (2025)

Counterfactual Inference for Eliminating Sentiment Bias in Recommender Systems
by: Pan, Le, et al.
Published: (2025)

CRAB: Codebook Rebalancing for Bias Mitigation in Generative Recommendation
by: Fan, Zezhong, et al.
Published: (2026)

Popularity-Aware Alignment and Contrast for Mitigating Popularity Bias
by: Cai, Miaomiao, et al.
Published: (2024)

Search and Society: Reimagining Information Access for Radical Futures
by: Mitra, Bhaskar
Published: (2024)