ChinaTravel: An Open-Ended Travel Planning Benchmark with Compositional Constraint Validation for Language Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918473211314176 |
|---|---|
| author | Shao, Jie-Jing Zhang, Bo-Wen Yang, Xiao-Wen Chen, Baizhi Han, Si-Yu Pang, Jinghao Wei, Wen-Da Cai, Guohao Dong, Zhenhua Guo, Lan-Zhe Li, Yu-Feng |
| author_facet | Shao, Jie-Jing Zhang, Bo-Wen Yang, Xiao-Wen Chen, Baizhi Han, Si-Yu Pang, Jinghao Wei, Wen-Da Cai, Guohao Dong, Zhenhua Guo, Lan-Zhe Li, Yu-Feng |
| contents | Travel planning stands out among real-world applications of \emph{Language Agents} because it couples significant practical demand with a rigorous constraint-satisfaction challenge. However, existing benchmarks primarily operate on a slot-filling paradigm, restricting agents to synthetic queries with pre-defined constraint menus, which fails to capture the open-ended nature of natural language interaction, where user requirements are compositional, diverse, and often implicitly expressed. To address this gap, we introduce \emph{ChinaTravel}, with four key contributions: 1) a practical sandbox aligned with the multi-day, multi-POI travel planning, 2) a compositionally generalizable domain-specific language (DSL) for scalable evaluation, covering feasibility, constraint satisfaction, and preference comparison 3) an open-ended dataset that integrates diverse travel requirements and implicit intent from 1154 human participants, and 4) fine-grained analysis reveal the potential of neuro-symbolic agents in travel planning, achieving a 37.0% constraint satisfaction rate on human queries, a 10 \times improvement over purely neural models, yet highlighting significant challenges in compositional generalization. Overall, ChinaTravel provides a foundation for advancing language agents through compositional constraint validation in complex, real-world planning scenarios. Project Page: https://www.lamda.nju.edu.cn/shaojj/ChinaTravel/index.html |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13682 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ChinaTravel: An Open-Ended Travel Planning Benchmark with Compositional Constraint Validation for Language Agents Shao, Jie-Jing Zhang, Bo-Wen Yang, Xiao-Wen Chen, Baizhi Han, Si-Yu Pang, Jinghao Wei, Wen-Da Cai, Guohao Dong, Zhenhua Guo, Lan-Zhe Li, Yu-Feng Artificial Intelligence Computation and Language Travel planning stands out among real-world applications of \emph{Language Agents} because it couples significant practical demand with a rigorous constraint-satisfaction challenge. However, existing benchmarks primarily operate on a slot-filling paradigm, restricting agents to synthetic queries with pre-defined constraint menus, which fails to capture the open-ended nature of natural language interaction, where user requirements are compositional, diverse, and often implicitly expressed. To address this gap, we introduce \emph{ChinaTravel}, with four key contributions: 1) a practical sandbox aligned with the multi-day, multi-POI travel planning, 2) a compositionally generalizable domain-specific language (DSL) for scalable evaluation, covering feasibility, constraint satisfaction, and preference comparison 3) an open-ended dataset that integrates diverse travel requirements and implicit intent from 1154 human participants, and 4) fine-grained analysis reveal the potential of neuro-symbolic agents in travel planning, achieving a 37.0% constraint satisfaction rate on human queries, a 10 \times improvement over purely neural models, yet highlighting significant challenges in compositional generalization. Overall, ChinaTravel provides a foundation for advancing language agents through compositional constraint validation in complex, real-world planning scenarios. Project Page: https://www.lamda.nju.edu.cn/shaojj/ChinaTravel/index.html |
| title | ChinaTravel: An Open-Ended Travel Planning Benchmark with Compositional Constraint Validation for Language Agents |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2412.13682 |