Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2501.17195 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910803138969600 |
|---|---|
| author | Alexandru, Andrei Calvi, Antonia Broomfield, Henry Golden, Jackson Dai, Kyle Leys, Mathias Burger, Maurice Bartolo, Max Engeler, Roman Pisupati, Sashank Drane, Toby Park, Young Sun |
| author_facet | Alexandru, Andrei Calvi, Antonia Broomfield, Henry Golden, Jackson Dai, Kyle Leys, Mathias Burger, Maurice Bartolo, Max Engeler, Roman Pisupati, Sashank Drane, Toby Park, Young Sun |
| contents | We introduce Atla Selene Mini, a state-of-the-art small language model-as-a-judge (SLMJ). Selene Mini is a general-purpose evaluator that outperforms the best SLMJs and GPT-4o-mini on overall performance across 11 out-of-distribution benchmarks, spanning absolute scoring, classification, and pairwise preference tasks. It is the highest-scoring 8B generative model on RewardBench, surpassing strong baselines like GPT-4o and specialized judges. To achieve this, we develop a principled data curation strategy that augments public datasets with synthetically generated critiques and ensures high quality through filtering and dataset ablations. We train our model on a combined direct preference optimization (DPO) and supervised fine-tuning (SFT) loss, and produce a highly promptable evaluator that excels in real-world scenarios. Selene Mini shows dramatically improved zero-shot agreement with human expert evaluations on financial and medical industry datasets. It is also robust to variations in prompt format. Preliminary results indicate that Selene Mini is the top-ranking evaluator in a live, community-driven Judge Arena. We release the model weights on HuggingFace (https://hf.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B) and Ollama to encourage widespread community adoption. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_17195 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Atla Selene Mini: A General Purpose Evaluation Model Alexandru, Andrei Calvi, Antonia Broomfield, Henry Golden, Jackson Dai, Kyle Leys, Mathias Burger, Maurice Bartolo, Max Engeler, Roman Pisupati, Sashank Drane, Toby Park, Young Sun Computation and Language Artificial Intelligence We introduce Atla Selene Mini, a state-of-the-art small language model-as-a-judge (SLMJ). Selene Mini is a general-purpose evaluator that outperforms the best SLMJs and GPT-4o-mini on overall performance across 11 out-of-distribution benchmarks, spanning absolute scoring, classification, and pairwise preference tasks. It is the highest-scoring 8B generative model on RewardBench, surpassing strong baselines like GPT-4o and specialized judges. To achieve this, we develop a principled data curation strategy that augments public datasets with synthetically generated critiques and ensures high quality through filtering and dataset ablations. We train our model on a combined direct preference optimization (DPO) and supervised fine-tuning (SFT) loss, and produce a highly promptable evaluator that excels in real-world scenarios. Selene Mini shows dramatically improved zero-shot agreement with human expert evaluations on financial and medical industry datasets. It is also robust to variations in prompt format. Preliminary results indicate that Selene Mini is the top-ranking evaluator in a live, community-driven Judge Arena. We release the model weights on HuggingFace (https://hf.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B) and Ollama to encourage widespread community adoption. |
| title | Atla Selene Mini: A General Purpose Evaluation Model |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.17195 |