StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917962708942848 |
|---|---|
| author | Chen, Hailin Jiao, Fangkai Ravaut, Mathieu Farruque, Nawshad Nguyen, Xuan Phi Qin, Chengwei Dey, Manan Ding, Bosheng Xiong, Caiming Joty, Shafiq Zhou, Yingbo |
| author_facet | Chen, Hailin Jiao, Fangkai Ravaut, Mathieu Farruque, Nawshad Nguyen, Xuan Phi Qin, Chengwei Dey, Manan Ding, Bosheng Xiong, Caiming Joty, Shafiq Zhou, Yingbo |
| contents | The rapid advancement of large language models (LLMs) demands robust, unbiased, and scalable evaluation methods. However, human annotations are costly to scale, model-based evaluations are susceptible to stylistic biases, and target-answer-based benchmarks are vulnerable to data contamination and cheating. To address these limitations, we propose StructTest, a novel benchmark that evaluates LLMs on their ability to follow compositional instructions and generate structured outputs, providing an unbiased, cost-effective, and difficult-to-cheat evaluation framework. Assessments are conducted deterministically using a rule-based evaluator, which can be easily extended to new tasks and datasets. By testing structured outputs across diverse domains including Summarization, Code, HTML, and Math, and evaluating 17 popular LLMs, we demonstrate that StructTest remains challenging even for top-performing models like Deepseek-V3/R1 and GPT-4o, establishing it as a robust proxy for measuring reasoning capabilities. We believe StructTest offers a critical and complementary approach to achieving objective and comprehensive model evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_18011 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs Chen, Hailin Jiao, Fangkai Ravaut, Mathieu Farruque, Nawshad Nguyen, Xuan Phi Qin, Chengwei Dey, Manan Ding, Bosheng Xiong, Caiming Joty, Shafiq Zhou, Yingbo Computation and Language The rapid advancement of large language models (LLMs) demands robust, unbiased, and scalable evaluation methods. However, human annotations are costly to scale, model-based evaluations are susceptible to stylistic biases, and target-answer-based benchmarks are vulnerable to data contamination and cheating. To address these limitations, we propose StructTest, a novel benchmark that evaluates LLMs on their ability to follow compositional instructions and generate structured outputs, providing an unbiased, cost-effective, and difficult-to-cheat evaluation framework. Assessments are conducted deterministically using a rule-based evaluator, which can be easily extended to new tasks and datasets. By testing structured outputs across diverse domains including Summarization, Code, HTML, and Math, and evaluating 17 popular LLMs, we demonstrate that StructTest remains challenging even for top-performing models like Deepseek-V3/R1 and GPT-4o, establishing it as a robust proxy for measuring reasoning capabilities. We believe StructTest offers a critical and complementary approach to achieving objective and comprehensive model evaluation. |
| title | StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.18011 |