Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.16575 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912285558046720 |
|---|---|
| author | Hu, Bo Yuan, Han Pandelea, Vlad Luo, Wuqiong Zhao, Yingzhu Ma, Zheng |
| author_facet | Hu, Bo Yuan, Han Pandelea, Vlad Luo, Wuqiong Zhao, Yingzhu Ma, Zheng |
| contents | The rapid advancement of large language models (LLMs) has sparked widespread adoption across diverse applications, making robust evaluation frameworks crucial for assessing their performance. While conventional evaluation metrics remain applicable for shorter texts, their efficacy diminishes when evaluating the quality of long-form answers. This limitation is particularly critical in real-world scenarios involving extended questions, extensive context, and long-form answers, such as financial analysis or regulatory compliance. In this paper, we use a practical financial use case to illustrate applications that handle "long question-context-answer triplets". We construct a real-world financial dataset comprising long triplets and demonstrate the inadequacies of traditional metrics. To address this, we propose an effective Extract, Match, and Score (EMS) evaluation approach tailored to the complexities of long-form LLMs' outputs, providing practitioners with a reliable methodology for assessing LLMs' performance in complex real-world scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_16575 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysis Hu, Bo Yuan, Han Pandelea, Vlad Luo, Wuqiong Zhao, Yingzhu Ma, Zheng Computation and Language Artificial Intelligence The rapid advancement of large language models (LLMs) has sparked widespread adoption across diverse applications, making robust evaluation frameworks crucial for assessing their performance. While conventional evaluation metrics remain applicable for shorter texts, their efficacy diminishes when evaluating the quality of long-form answers. This limitation is particularly critical in real-world scenarios involving extended questions, extensive context, and long-form answers, such as financial analysis or regulatory compliance. In this paper, we use a practical financial use case to illustrate applications that handle "long question-context-answer triplets". We construct a real-world financial dataset comprising long triplets and demonstrate the inadequacies of traditional metrics. To address this, we propose an effective Extract, Match, and Score (EMS) evaluation approach tailored to the complexities of long-form LLMs' outputs, providing practitioners with a reliable methodology for assessing LLMs' performance in complex real-world scenarios. |
| title | Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysis |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.16575 |