LLMs as Meta-Reviewers' Assistants: A Case Study
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910819308011520 |
|---|---|
| author | Hossain, Eftekhar Sinha, Sanjeev Kumar Bansal, Naman Knipper, Alex Sarkar, Souvika Salvador, John Mahajan, Yash Guttikonda, Sri Akter, Mousumi Hassan, Md. Mahadi Freestone, Matthew Williams Jr., Matthew C. Feng, Dongji Karmaker, Santu |
| author_facet | Hossain, Eftekhar Sinha, Sanjeev Kumar Bansal, Naman Knipper, Alex Sarkar, Souvika Salvador, John Mahajan, Yash Guttikonda, Sri Akter, Mousumi Hassan, Md. Mahadi Freestone, Matthew Williams Jr., Matthew C. Feng, Dongji Karmaker, Santu |
| contents | One of the most important yet onerous tasks in the academic peer-reviewing process is composing meta-reviews, which involves assimilating diverse opinions from multiple expert peers, formulating one's self-judgment as a senior expert, and then summarizing all these perspectives into a concise holistic overview to make an overall recommendation. This process is time-consuming and can be compromised by human factors like fatigue, inconsistency, missing tiny details, etc. Given the latest major developments in Large Language Models (LLMs), it is very compelling to rigorously study whether LLMs can help metareviewers perform this important task better. In this paper, we perform a case study with three popular LLMs, i.e., GPT-3.5, LLaMA2, and PaLM2, to assist meta-reviewers in better comprehending multiple experts perspectives by generating a controlled multi-perspective summary (MPS) of their opinions. To achieve this, we prompt three LLMs with different types/levels of prompts based on the recently proposed TELeR taxonomy. Finally, we perform a detailed qualitative study of the MPSs generated by the LLMs and report our findings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_15589 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLMs as Meta-Reviewers' Assistants: A Case Study Hossain, Eftekhar Sinha, Sanjeev Kumar Bansal, Naman Knipper, Alex Sarkar, Souvika Salvador, John Mahajan, Yash Guttikonda, Sri Akter, Mousumi Hassan, Md. Mahadi Freestone, Matthew Williams Jr., Matthew C. Feng, Dongji Karmaker, Santu Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing I.2.7 One of the most important yet onerous tasks in the academic peer-reviewing process is composing meta-reviews, which involves assimilating diverse opinions from multiple expert peers, formulating one's self-judgment as a senior expert, and then summarizing all these perspectives into a concise holistic overview to make an overall recommendation. This process is time-consuming and can be compromised by human factors like fatigue, inconsistency, missing tiny details, etc. Given the latest major developments in Large Language Models (LLMs), it is very compelling to rigorously study whether LLMs can help metareviewers perform this important task better. In this paper, we perform a case study with three popular LLMs, i.e., GPT-3.5, LLaMA2, and PaLM2, to assist meta-reviewers in better comprehending multiple experts perspectives by generating a controlled multi-perspective summary (MPS) of their opinions. To achieve this, we prompt three LLMs with different types/levels of prompts based on the recently proposed TELeR taxonomy. Finally, we perform a detailed qualitative study of the MPSs generated by the LLMs and report our findings. |
| title | LLMs as Meta-Reviewers' Assistants: A Case Study |
| topic | Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing I.2.7 |
| url | https://arxiv.org/abs/2402.15589 |