Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2511.09997 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912706512027648 |
|---|---|
| author | Huang, Yu-Shiang Lee, Yun-Yu Chou, Tzu-Hsin Lin, Che Wang, Chuan-Ju |
| author_facet | Huang, Yu-Shiang Lee, Yun-Yu Chou, Tzu-Hsin Lin, Che Wang, Chuan-Ju |
| contents | BERTScore has become a widely adopted metric for evaluating semantic similarity between natural language sentences. However, we identify a critical limitation: BERTScore exhibits low sensitivity to numerical variation, a significant weakness in finance where numerical precision directly affects meaning (e.g., distinguishing a 2% gain from a 20% loss). We introduce FinNuE, a diagnostic dataset constructed with controlled numerical perturbations across earnings calls, regulatory filings, social media, and news articles. Using FinNuE, demonstrate that BERTScore fails to distinguish semantically critical numerical differences, often assigning high similarity scores to financially divergent text pairs. Our findings reveal fundamental limitations of embedding-based metrics for finance and motivate numerically-aware evaluation frameworks for financial NLP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_09997 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FinNuE: Exposing the Risks of Using BERTScore for Numerical Semantic Evaluation in Finance Huang, Yu-Shiang Lee, Yun-Yu Chou, Tzu-Hsin Lin, Che Wang, Chuan-Ju Computation and Language BERTScore has become a widely adopted metric for evaluating semantic similarity between natural language sentences. However, we identify a critical limitation: BERTScore exhibits low sensitivity to numerical variation, a significant weakness in finance where numerical precision directly affects meaning (e.g., distinguishing a 2% gain from a 20% loss). We introduce FinNuE, a diagnostic dataset constructed with controlled numerical perturbations across earnings calls, regulatory filings, social media, and news articles. Using FinNuE, demonstrate that BERTScore fails to distinguish semantically critical numerical differences, often assigning high similarity scores to financially divergent text pairs. Our findings reveal fundamental limitations of embedding-based metrics for finance and motivate numerically-aware evaluation frameworks for financial NLP. |
| title | FinNuE: Exposing the Risks of Using BERTScore for Numerical Semantic Evaluation in Finance |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.09997 |