Adaptively profiling models with task elicitation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911176320876544 |
|---|---|
| author | Brown, Davis Balehannina, Prithvi Jin, Helen Havaldar, Shreya Hassani, Hamed Wong, Eric |
| author_facet | Brown, Davis Balehannina, Prithvi Jin, Helen Havaldar, Shreya Hassani, Hamed Wong, Eric |
| contents | Language model evaluations often fail to characterize consequential failure modes, forcing experts to inspect outputs and build new benchmarks. We introduce task elicitation, a method that automatically builds new evaluations to profile model behavior. Task elicitation finds hundreds of natural-language tasks -- an order of magnitude more than prior work -- where frontier models exhibit systematic failures, in domains ranging from forecasting to online harassment. For example, we find that Sonnet 3.5 over-associates quantum computing and AGI and that o3-mini is prone to hallucination when fabrications are repeated in-context. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01986 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adaptively profiling models with task elicitation Brown, Davis Balehannina, Prithvi Jin, Helen Havaldar, Shreya Hassani, Hamed Wong, Eric Computation and Language Artificial Intelligence Machine Learning Language model evaluations often fail to characterize consequential failure modes, forcing experts to inspect outputs and build new benchmarks. We introduce task elicitation, a method that automatically builds new evaluations to profile model behavior. Task elicitation finds hundreds of natural-language tasks -- an order of magnitude more than prior work -- where frontier models exhibit systematic failures, in domains ranging from forecasting to online harassment. For example, we find that Sonnet 3.5 over-associates quantum computing and AGI and that o3-mini is prone to hallucination when fabrications are repeated in-context. |
| title | Adaptively profiling models with task elicitation |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2503.01986 |