Is Your LLM Outdated? A Deep Look at Temporal Generalization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916818643320832 |
|---|---|
| author | Zhu, Chenghao Chen, Nuo Gao, Yufei Zhang, Yunyi Tiwari, Prayag Wang, Benyou |
| author_facet | Zhu, Chenghao Chen, Nuo Gao, Yufei Zhang, Yunyi Tiwari, Prayag Wang, Benyou |
| contents | The rapid advancement of Large Language Models (LLMs) has led to the development of benchmarks that consider temporal dynamics, however, there remains a gap in understanding how well these models can generalize across temporal contexts due to the inherent dynamic nature of language and information. This paper introduces the concept of temporal generalization in LLMs, including bias in past and future generalizations. Then we introduce FreshBench, a new evaluation framework that employs fresh text and event prediction for assessing LLMs' temporal adaptability, ensuring the evaluation process free from data leakage and subjective bias. The experiment shows significant temporal biases and a decline in performance over time. Our findings reveal that powerful models, while initially superior, tend to decline more rapidly in future generalization. Additionally, powerful open-source models demonstrate better long-term adaptability compared to their closed-source counterparts. Our code is available at https://github.com/FreedomIntelligence/FreshBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_08460 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Is Your LLM Outdated? A Deep Look at Temporal Generalization Zhu, Chenghao Chen, Nuo Gao, Yufei Zhang, Yunyi Tiwari, Prayag Wang, Benyou Computation and Language Artificial Intelligence The rapid advancement of Large Language Models (LLMs) has led to the development of benchmarks that consider temporal dynamics, however, there remains a gap in understanding how well these models can generalize across temporal contexts due to the inherent dynamic nature of language and information. This paper introduces the concept of temporal generalization in LLMs, including bias in past and future generalizations. Then we introduce FreshBench, a new evaluation framework that employs fresh text and event prediction for assessing LLMs' temporal adaptability, ensuring the evaluation process free from data leakage and subjective bias. The experiment shows significant temporal biases and a decline in performance over time. Our findings reveal that powerful models, while initially superior, tend to decline more rapidly in future generalization. Additionally, powerful open-source models demonstrate better long-term adaptability compared to their closed-source counterparts. Our code is available at https://github.com/FreedomIntelligence/FreshBench. |
| title | Is Your LLM Outdated? A Deep Look at Temporal Generalization |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.08460 |