PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929602056683520 |
|---|---|
| author | Friedland, Gerald Huang, Xin Cui, Yueying Kapoor, Vishaal Khetan, Ashish Das, Sanjiv |
| author_facet | Friedland, Gerald Huang, Xin Cui, Yueying Kapoor, Vishaal Khetan, Ashish Das, Sanjiv |
| contents | We propose PPLqa, an easy to compute, language independent, information-theoretic metric to measure the quality of responses of generative Large Language Models (LLMs) in an unsupervised way, without requiring ground truth annotations or human supervision. The method and metric enables users to rank generative language models for quality of responses, so as to make a selection of the best model for a given task. Our single metric assesses LLMs with an approach that subsumes, but is not explicitly based on, coherence and fluency (quality of writing) and relevance and consistency (appropriateness of response) to the query. PPLqa performs as well as other related metrics, and works better with long-form Q\&A. Thus, PPLqa enables bypassing the lengthy annotation process required for ground truth evaluations, and it also correlates well with human and LLM rankings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_15320 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models Friedland, Gerald Huang, Xin Cui, Yueying Kapoor, Vishaal Khetan, Ashish Das, Sanjiv Computation and Language Artificial Intelligence I.2.m; E.4 We propose PPLqa, an easy to compute, language independent, information-theoretic metric to measure the quality of responses of generative Large Language Models (LLMs) in an unsupervised way, without requiring ground truth annotations or human supervision. The method and metric enables users to rank generative language models for quality of responses, so as to make a selection of the best model for a given task. Our single metric assesses LLMs with an approach that subsumes, but is not explicitly based on, coherence and fluency (quality of writing) and relevance and consistency (appropriateness of response) to the query. PPLqa performs as well as other related metrics, and works better with long-form Q\&A. Thus, PPLqa enables bypassing the lengthy annotation process required for ground truth evaluations, and it also correlates well with human and LLM rankings. |
| title | PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models |
| topic | Computation and Language Artificial Intelligence I.2.m; E.4 |
| url | https://arxiv.org/abs/2411.15320 |