Ebisu: Benchmarking Large Language Models in Japanese Finance
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914300409413632 |
|---|---|
| author | Peng, Xueqing Xiang, Ruoyu Zhang, Fan Song, Mingzi Jiang, Mingyang Wang, Yan Qian, Lingfei Hara, Taiki Guo, Yuqing Huang, Jimin Tsujii, Junichi Ananiadou, Sophia |
| author_facet | Peng, Xueqing Xiang, Ruoyu Zhang, Fan Song, Mingzi Jiang, Mingyang Wang, Yan Qian, Lingfei Hara, Taiki Guo, Yuqing Huang, Jimin Tsujii, Junichi Ananiadou, Sophia |
| contents | Japanese finance combines agglutinative, head-final linguistic structure, mixed writing systems, and high-context communication norms that rely on indirect expression and implicit commitment, posing a substantial challenge for LLMs. We introduce Ebisu, a benchmark for native Japanese financial language understanding, comprising two linguistically and culturally grounded, expert-annotated tasks: JF-ICR, which evaluates implicit commitment and refusal recognition in investor-facing Q&A, and JF-TE, which assesses hierarchical extraction and ranking of nested financial terminology from professional disclosures. We evaluate a diverse set of open-source and proprietary LLMs spanning general-purpose, Japanese-adapted, and financial models. Results show that even state-of-the-art systems struggle on both tasks. While increased model scale yields limited improvements, language- and domain-specific adaptation does not reliably improve performance, leaving substantial gaps unresolved. Ebisu provides a focused benchmark for advancing linguistically and culturally grounded financial NLP. All datasets and evaluation scripts are publicly released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01479 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Ebisu: Benchmarking Large Language Models in Japanese Finance Peng, Xueqing Xiang, Ruoyu Zhang, Fan Song, Mingzi Jiang, Mingyang Wang, Yan Qian, Lingfei Hara, Taiki Guo, Yuqing Huang, Jimin Tsujii, Junichi Ananiadou, Sophia Computation and Language Japanese finance combines agglutinative, head-final linguistic structure, mixed writing systems, and high-context communication norms that rely on indirect expression and implicit commitment, posing a substantial challenge for LLMs. We introduce Ebisu, a benchmark for native Japanese financial language understanding, comprising two linguistically and culturally grounded, expert-annotated tasks: JF-ICR, which evaluates implicit commitment and refusal recognition in investor-facing Q&A, and JF-TE, which assesses hierarchical extraction and ranking of nested financial terminology from professional disclosures. We evaluate a diverse set of open-source and proprietary LLMs spanning general-purpose, Japanese-adapted, and financial models. Results show that even state-of-the-art systems struggle on both tasks. While increased model scale yields limited improvements, language- and domain-specific adaptation does not reliably improve performance, leaving substantial gaps unresolved. Ebisu provides a focused benchmark for advancing linguistically and culturally grounded financial NLP. All datasets and evaluation scripts are publicly released. |
| title | Ebisu: Benchmarking Large Language Models in Japanese Finance |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.01479 |