Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917252120444928 |
|---|---|
| author | Epstein, Elliot L. Dwaraknath, Rajat Winnicki, John Sornwanee, Thanawat |
| author_facet | Epstein, Elliot L. Dwaraknath, Rajat Winnicki, John Sornwanee, Thanawat |
| contents | This paper argues that large ML conferences should allocate marginal review capacity primarily to papers near the acceptance boundary, rather than spreading extra reviews via random or affinity-driven heuristics. We propose using LLM-based comparative ranking (via pairwise comparisons and a Bradley--Terry model) to identify a borderline band \emph{before} human reviewing and to allocate \emph{marginal} reviewer capacity at assignment time. Concretely, given a venue-specific minimum review target (e.g., 3 or 4), we use this signal to decide which papers receive one additional review (e.g., a 4th or 5th), without conditioning on any human reviews and without using LLM outputs for accept/reject. We provide a simple expected-impact calculation in terms of (i) the overlap between the predicted and true borderline sets ($ρ$) and (ii) the incremental value of an extra review near the boundary ($Δ$), and we provide retrospective proxies to estimate these quantities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_06078 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking Epstein, Elliot L. Dwaraknath, Rajat Winnicki, John Sornwanee, Thanawat Digital Libraries Artificial Intelligence Machine Learning I.2.7 This paper argues that large ML conferences should allocate marginal review capacity primarily to papers near the acceptance boundary, rather than spreading extra reviews via random or affinity-driven heuristics. We propose using LLM-based comparative ranking (via pairwise comparisons and a Bradley--Terry model) to identify a borderline band \emph{before} human reviewing and to allocate \emph{marginal} reviewer capacity at assignment time. Concretely, given a venue-specific minimum review target (e.g., 3 or 4), we use this signal to decide which papers receive one additional review (e.g., a 4th or 5th), without conditioning on any human reviews and without using LLM outputs for accept/reject. We provide a simple expected-impact calculation in terms of (i) the overlap between the predicted and true borderline sets ($ρ$) and (ii) the incremental value of an extra review near the boundary ($Δ$), and we provide retrospective proxies to estimate these quantities. |
| title | Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking |
| topic | Digital Libraries Artificial Intelligence Machine Learning I.2.7 |
| url | https://arxiv.org/abs/2602.06078 |