Exploring Precision and Recall to assess the quality and diversity of LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929372470968320 |
|---|---|
| author | Bronnec, Florian Le Verine, Alexandre Negrevergne, Benjamin Chevaleyre, Yann Allauzen, Alexandre |
| author_facet | Bronnec, Florian Le Verine, Alexandre Negrevergne, Benjamin Chevaleyre, Yann Allauzen, Alexandre |
| contents | We introduce a novel evaluation framework for Large Language Models (LLMs) such as \textsc{Llama-2} and \textsc{Mistral}, focusing on importing Precision and Recall metrics from image generation to text generation. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora. By conducting a comprehensive evaluation of state-of-the-art language models, the study reveals new insights into their performance on open-ended generation tasks, which are not adequately captured by traditional benchmarks. The findings highlight a trade-off between the quality and diversity of generated samples, particularly when models are fine-tuned on instruction dataset or with human feedback. This work extends the toolkit for distribution-based NLP evaluation, offering insights into the practical capabilities and challenges that current LLMs face in generating diverse and high-quality text. We release our code and data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_10693 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring Precision and Recall to assess the quality and diversity of LLMs Bronnec, Florian Le Verine, Alexandre Negrevergne, Benjamin Chevaleyre, Yann Allauzen, Alexandre Computation and Language Machine Learning We introduce a novel evaluation framework for Large Language Models (LLMs) such as \textsc{Llama-2} and \textsc{Mistral}, focusing on importing Precision and Recall metrics from image generation to text generation. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora. By conducting a comprehensive evaluation of state-of-the-art language models, the study reveals new insights into their performance on open-ended generation tasks, which are not adequately captured by traditional benchmarks. The findings highlight a trade-off between the quality and diversity of generated samples, particularly when models are fine-tuned on instruction dataset or with human feedback. This work extends the toolkit for distribution-based NLP evaluation, offering insights into the practical capabilities and challenges that current LLMs face in generating diverse and high-quality text. We release our code and data. |
| title | Exploring Precision and Recall to assess the quality and diversity of LLMs |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2402.10693 |