Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.01986 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911176320876544 |
|---|---|
| author | Brown, Davis Balehannina, Prithvi Jin, Helen Havaldar, Shreya Hassani, Hamed Wong, Eric |
| author_facet | Brown, Davis Balehannina, Prithvi Jin, Helen Havaldar, Shreya Hassani, Hamed Wong, Eric |
| contents | Language model evaluations often fail to characterize consequential failure modes, forcing experts to inspect outputs and build new benchmarks. We introduce task elicitation, a method that automatically builds new evaluations to profile model behavior. Task elicitation finds hundreds of natural-language tasks -- an order of magnitude more than prior work -- where frontier models exhibit systematic failures, in domains ranging from forecasting to online harassment. For example, we find that Sonnet 3.5 over-associates quantum computing and AGI and that o3-mini is prone to hallucination when fabrications are repeated in-context. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01986 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adaptively profiling models with task elicitation Brown, Davis Balehannina, Prithvi Jin, Helen Havaldar, Shreya Hassani, Hamed Wong, Eric Computation and Language Artificial Intelligence Machine Learning Language model evaluations often fail to characterize consequential failure modes, forcing experts to inspect outputs and build new benchmarks. We introduce task elicitation, a method that automatically builds new evaluations to profile model behavior. Task elicitation finds hundreds of natural-language tasks -- an order of magnitude more than prior work -- where frontier models exhibit systematic failures, in domains ranging from forecasting to online harassment. For example, we find that Sonnet 3.5 over-associates quantum computing and AGI and that o3-mini is prone to hallucination when fabrications are repeated in-context. |
| title | Adaptively profiling models with task elicitation |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2503.01986 |