Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916377325993984 |
|---|---|
| author | Yang, Blair Cui, Fuyang Paster, Keiran Ba, Jimmy Vaezipoor, Pashootan Pitis, Silviu Zhang, Michael R. |
| author_facet | Yang, Blair Cui, Fuyang Paster, Keiran Ba, Jimmy Vaezipoor, Pashootan Pitis, Silviu Zhang, Michael R. |
| contents | The rapid development and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities. We propose report cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics. We develop a framework to evaluate report cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating report cards without human supervision and explore its efficacy by ablating various design choices. Through experimentation with popular LLMs, we demonstrate that report cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_00844 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries Yang, Blair Cui, Fuyang Paster, Keiran Ba, Jimmy Vaezipoor, Pashootan Pitis, Silviu Zhang, Michael R. Machine Learning Artificial Intelligence Computation and Language The rapid development and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities. We propose report cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics. We develop a framework to evaluate report cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating report cards without human supervision and explore its efficacy by ablating various design choices. Through experimentation with popular LLMs, we demonstrate that report cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs. |
| title | Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2409.00844 |