Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.21329 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910063030960128 |
|---|---|
| author | Li, Xiaozhe Lyu, Tianyi Yang, Siyi Yang, Yizhao Gong, Yuxi Huang, Jinxuan Zhang, Ligao Huang, Zhuoyi Liu, Qingwen |
| author_facet | Li, Xiaozhe Lyu, Tianyi Yang, Siyi Yang, Yizhao Gong, Yuxi Huang, Jinxuan Zhang, Ligao Huang, Zhuoyi Liu, Qingwen |
| contents | Understanding human intent is a high-level cognitive challenge for Large Language Models (LLMs), requiring sophisticated reasoning over noisy, conflicting, and non-linear discourse. While LLMs excel at following individual instructions, their ability to distill Collective Intent - the process of extracting consensus, resolving contradictions, and inferring latent trends from multi-source public discussions - remains largely unexplored. To bridge this gap, we introduce COIN-BENCH, a dynamic, real-world, live-updating benchmark specifically designed to evaluate LLMs on collective intent understanding within the consumer domain. Unlike traditional benchmarks that focus on transactional outcomes, COIN-BENCH operationalizes intent as a hierarchical cognitive structure, ranging from explicit scenarios to deep causal reasoning. We implement a robust evaluation pipeline that combines a rule-based method with an LLM-as-the-Judge approach. This framework incorporates COIN-TREE for hierarchical cognitive structuring and retrieval-augmented verification (COIN-RAG) to ensure expert-level precision in analyzing raw, collective human discussions. An extensive evaluation of 20 state-of-the-art LLMs across four dimensions - depth, breadth, informativeness, and correctness - reveals that while current models can handle surface-level aggregation, they still struggle with the analytical depth required for complex intent synthesis. COIN-BENCH establishes a new standard for advancing LLMs from passive instruction followers to expert-level analytical agents capable of deciphering the collective voice of the real world. See our project page on COIN-BENCH. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_21329 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | COINBench: Moving Beyond Individual Perspectives to Collective Intent Understanding Li, Xiaozhe Lyu, Tianyi Yang, Siyi Yang, Yizhao Gong, Yuxi Huang, Jinxuan Zhang, Ligao Huang, Zhuoyi Liu, Qingwen Information Retrieval Artificial Intelligence Understanding human intent is a high-level cognitive challenge for Large Language Models (LLMs), requiring sophisticated reasoning over noisy, conflicting, and non-linear discourse. While LLMs excel at following individual instructions, their ability to distill Collective Intent - the process of extracting consensus, resolving contradictions, and inferring latent trends from multi-source public discussions - remains largely unexplored. To bridge this gap, we introduce COIN-BENCH, a dynamic, real-world, live-updating benchmark specifically designed to evaluate LLMs on collective intent understanding within the consumer domain. Unlike traditional benchmarks that focus on transactional outcomes, COIN-BENCH operationalizes intent as a hierarchical cognitive structure, ranging from explicit scenarios to deep causal reasoning. We implement a robust evaluation pipeline that combines a rule-based method with an LLM-as-the-Judge approach. This framework incorporates COIN-TREE for hierarchical cognitive structuring and retrieval-augmented verification (COIN-RAG) to ensure expert-level precision in analyzing raw, collective human discussions. An extensive evaluation of 20 state-of-the-art LLMs across four dimensions - depth, breadth, informativeness, and correctness - reveals that while current models can handle surface-level aggregation, they still struggle with the analytical depth required for complex intent synthesis. COIN-BENCH establishes a new standard for advancing LLMs from passive instruction followers to expert-level analytical agents capable of deciphering the collective voice of the real world. See our project page on COIN-BENCH. |
| title | COINBench: Moving Beyond Individual Perspectives to Collective Intent Understanding |
| topic | Information Retrieval Artificial Intelligence |
| url | https://arxiv.org/abs/2603.21329 |