Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.06089 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913538013921280 |
|---|---|
| author | Ziems, Noah Zhang, Zhihan Jiang, Meng |
| author_facet | Ziems, Noah Zhang, Zhihan Jiang, Meng |
| contents | Evaluating the ability of large language models (LLMs) to follow complex human-written instructions is essential for their deployment in real-world applications. While benchmarks like Chatbot Arena use human judges to assess model performance, they are resource-intensive and time-consuming. Alternative methods using LLMs as judges, such as AlpacaEval, MT Bench, WildBench, and InFoBench offer improvements but still do not capture that certain complex instruction aspects are more important than others to follow.
To address this gap, we propose a novel evaluation metric, \textsc{TOWER}, that incorporates human-judged importance into the assessment of complex instruction following. We show that human annotators agree with tree-based representations of these complex instructions nearly as much as they agree with other human annotators. We release tree-based annotations of the InFoBench dataset and the corresponding evaluation code to facilitate future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_06089 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TOWER: Tree Organized Weighting for Evaluating Complex Instructions Ziems, Noah Zhang, Zhihan Jiang, Meng Computation and Language Artificial Intelligence Evaluating the ability of large language models (LLMs) to follow complex human-written instructions is essential for their deployment in real-world applications. While benchmarks like Chatbot Arena use human judges to assess model performance, they are resource-intensive and time-consuming. Alternative methods using LLMs as judges, such as AlpacaEval, MT Bench, WildBench, and InFoBench offer improvements but still do not capture that certain complex instruction aspects are more important than others to follow. To address this gap, we propose a novel evaluation metric, \textsc{TOWER}, that incorporates human-judged importance into the assessment of complex instruction following. We show that human annotators agree with tree-based representations of these complex instructions nearly as much as they agree with other human annotators. We release tree-based annotations of the InFoBench dataset and the corresponding evaluation code to facilitate future research. |
| title | TOWER: Tree Organized Weighting for Evaluating Complex Instructions |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.06089 |