MoZIP: A Multilingual Benchmark to Evaluate Large Language Models in Intellectual Property
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910343750483968 |
|---|---|
| author | Ni, Shiwen Tan, Minghuan Bai, Yuelin Niu, Fuqiang Yang, Min Zhang, Bowen Xu, Ruifeng Chen, Xiaojun Li, Chengming Hu, Xiping Li, Ye Fan, Jianping |
| author_facet | Ni, Shiwen Tan, Minghuan Bai, Yuelin Niu, Fuqiang Yang, Min Zhang, Bowen Xu, Ruifeng Chen, Xiaojun Li, Chengming Hu, Xiping Li, Ye Fan, Jianping |
| contents | Large language models (LLMs) have demonstrated impressive performance in various natural language processing (NLP) tasks. However, there is limited understanding of how well LLMs perform in specific domains (e.g, the intellectual property (IP) domain). In this paper, we contribute a new benchmark, the first Multilingual-oriented quiZ on Intellectual Property (MoZIP), for the evaluation of LLMs in the IP domain. The MoZIP benchmark includes three challenging tasks: IP multiple-choice quiz (IPQuiz), IP question answering (IPQA), and patent matching (PatentMatch). In addition, we also develop a new IP-oriented multilingual large language model (called MoZi), which is a BLOOMZ-based model that has been supervised fine-tuned with multilingual IP-related text data. We evaluate our proposed MoZi model and four well-known LLMs (i.e., BLOOMZ, BELLE, ChatGLM and ChatGPT) on the MoZIP benchmark. Experimental results demonstrate that MoZi outperforms BLOOMZ, BELLE and ChatGLM by a noticeable margin, while it had lower scores compared with ChatGPT. Notably, the performance of current LLMs on the MoZIP benchmark has much room for improvement, and even the most powerful ChatGPT does not reach the passing level. Our source code, data, and models are available at \url{https://github.com/AI-for-Science/MoZi}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_16389 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MoZIP: A Multilingual Benchmark to Evaluate Large Language Models in Intellectual Property Ni, Shiwen Tan, Minghuan Bai, Yuelin Niu, Fuqiang Yang, Min Zhang, Bowen Xu, Ruifeng Chen, Xiaojun Li, Chengming Hu, Xiping Li, Ye Fan, Jianping Computation and Language Artificial Intelligence Large language models (LLMs) have demonstrated impressive performance in various natural language processing (NLP) tasks. However, there is limited understanding of how well LLMs perform in specific domains (e.g, the intellectual property (IP) domain). In this paper, we contribute a new benchmark, the first Multilingual-oriented quiZ on Intellectual Property (MoZIP), for the evaluation of LLMs in the IP domain. The MoZIP benchmark includes three challenging tasks: IP multiple-choice quiz (IPQuiz), IP question answering (IPQA), and patent matching (PatentMatch). In addition, we also develop a new IP-oriented multilingual large language model (called MoZi), which is a BLOOMZ-based model that has been supervised fine-tuned with multilingual IP-related text data. We evaluate our proposed MoZi model and four well-known LLMs (i.e., BLOOMZ, BELLE, ChatGLM and ChatGPT) on the MoZIP benchmark. Experimental results demonstrate that MoZi outperforms BLOOMZ, BELLE and ChatGLM by a noticeable margin, while it had lower scores compared with ChatGPT. Notably, the performance of current LLMs on the MoZIP benchmark has much room for improvement, and even the most powerful ChatGPT does not reach the passing level. Our source code, data, and models are available at \url{https://github.com/AI-for-Science/MoZi}. |
| title | MoZIP: A Multilingual Benchmark to Evaluate Large Language Models in Intellectual Property |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.16389 |