The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909981846011904 |
|---|---|
| author | Ren, Richard Agarwal, Arunim Mazeika, Mantas Menghini, Cristina Vacareanu, Robert Kenstler, Brad Yang, Mick Barrass, Isabelle Gatti, Alice Yin, Xuwang Trevino, Eduardo Geralnik, Matias Khoja, Adam Lee, Dean Yue, Summer Hendrycks, Dan |
| author_facet | Ren, Richard Agarwal, Arunim Mazeika, Mantas Menghini, Cristina Vacareanu, Robert Kenstler, Brad Yang, Mick Barrass, Isabelle Gatti, Alice Yin, Xuwang Trevino, Eduardo Geralnik, Matias Khoja, Adam Lee, Dean Yue, Summer Hendrycks, Dan |
| contents | As large language models (LLMs) become more capable and agentic, the requirement for trust in their outputs grows significantly, yet at the same time concerns have been mounting that models may learn to lie in pursuit of their goals. To address these concerns, a body of work has emerged around the notion of "honesty" in LLMs, along with interventions aimed at mitigating deceptive behaviors. However, some benchmarks claiming to measure honesty in fact simply measure accuracy--the correctness of a model's beliefs--in disguise. Moreover, no benchmarks currently exist for directly measuring whether language models lie. In this work, we introduce a large-scale human-collected dataset for directly measuring lying, allowing us to disentangle accuracy from honesty. Across a diverse set of LLMs, we find that while larger models obtain higher accuracy on our benchmark, they do not become more honest. Surprisingly, most frontier LLMs obtain high scores on truthfulness benchmarks yet exhibit a substantial propensity to lie under pressure, resulting in low honesty scores on our benchmark. We find that simple methods, such as representation engineering interventions, can improve honesty. These results underscore the growing need for robust evaluations and effective interventions to ensure LLMs remain trustworthy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_03750 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems Ren, Richard Agarwal, Arunim Mazeika, Mantas Menghini, Cristina Vacareanu, Robert Kenstler, Brad Yang, Mick Barrass, Isabelle Gatti, Alice Yin, Xuwang Trevino, Eduardo Geralnik, Matias Khoja, Adam Lee, Dean Yue, Summer Hendrycks, Dan Machine Learning Artificial Intelligence Computation and Language Computers and Society As large language models (LLMs) become more capable and agentic, the requirement for trust in their outputs grows significantly, yet at the same time concerns have been mounting that models may learn to lie in pursuit of their goals. To address these concerns, a body of work has emerged around the notion of "honesty" in LLMs, along with interventions aimed at mitigating deceptive behaviors. However, some benchmarks claiming to measure honesty in fact simply measure accuracy--the correctness of a model's beliefs--in disguise. Moreover, no benchmarks currently exist for directly measuring whether language models lie. In this work, we introduce a large-scale human-collected dataset for directly measuring lying, allowing us to disentangle accuracy from honesty. Across a diverse set of LLMs, we find that while larger models obtain higher accuracy on our benchmark, they do not become more honest. Surprisingly, most frontier LLMs obtain high scores on truthfulness benchmarks yet exhibit a substantial propensity to lie under pressure, resulting in low honesty scores on our benchmark. We find that simple methods, such as representation engineering interventions, can improve honesty. These results underscore the growing need for robust evaluations and effective interventions to ensure LLMs remain trustworthy. |
| title | The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems |
| topic | Machine Learning Artificial Intelligence Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2503.03750 |