Image de couverture de livre

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Merrill, Mike A., Shaw, Alexander G., Carlini, Nicholas, Li, Boxuan, Raj, Harsh, Bercovich, Ivan, Shi, Lin, Shin, Jeong Yeon, Walshe, Thomas, Buchanan, E. Kelly, Shen, Junhong, Ye, Guanghao, Lin, Haowei, Poulos, Jason, Wang, Maoyu, Nezhurina, Marianna, Jitsev, Jenia, Lu, Di, Mastromichalakis, Orfeas Menis, Xu, Zhiwei, Chen, Zizhao, Liu, Yue, Zhang, Robert, Chen, Leon Liangyu, Kashyap, Anurag, Uslu, Jan-Lucas, Li, Jeffrey, Wu, Jianbo, Yan, Minghao, Bian, Song, Sharma, Vedang, Sun, Ke, Dillmann, Steven, Anand, Akshay, Lanpouthakoun, Andrew, Koopah, Bardia, Hu, Changran, Guha, Etash, Dreiman, Gabriel H. S., Zhu, Jiacheng, Krauth, Karl, Zhong, Li, Muennighoff, Niklas, Amanfu, Robert, Tan, Shangyin, Pimpalgaonkar, Shreyas, Aggarwal, Tushar, Lin, Xiangning, Lan, Xin, Zhao, Xuandong, Liang, Yiqing, Wang, Yuanli, Wang, Zilong, Zhou, Changzhi, Heineman, David, Liu, Hange, Trivedi, Harsh, Yang, John, Lin, Junhong, Shetty, Manish, Yang, Michael, Omi, Nabil, Raoof, Negin, Li, Shanda, Zhuo, Terry Yue, Lin, Wuwei, Dai, Yiwei, Wang, Yuxin, Chai, Wenhao, Zhou, Shang, Wahdany, Dariush, She, Ziyu, Hu, Jiaming, Dong, Zhikang, Zhu, Yuxuan, Cui, Sasha, Saiyed, Ahson, Kolbeinsson, Arinbjörn, Hu, Jesse, Rytting, Christopher Michael, Marten, Ryan, Wang, Yixin, Dimakis, Alex, Konwinski, Andy, Schmidt, Ludwig
Format:	Preprint
Publié:	2026
Sujets:	Software Engineering Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

_version_	1866912829931520000
author	Merrill, Mike A. Shaw, Alexander G. Carlini, Nicholas Li, Boxuan Raj, Harsh Bercovich, Ivan Shi, Lin Shin, Jeong Yeon Walshe, Thomas Buchanan, E. Kelly Shen, Junhong Ye, Guanghao Lin, Haowei Poulos, Jason Wang, Maoyu Nezhurina, Marianna Jitsev, Jenia Lu, Di Mastromichalakis, Orfeas Menis Xu, Zhiwei Chen, Zizhao Liu, Yue Zhang, Robert Chen, Leon Liangyu Kashyap, Anurag Uslu, Jan-Lucas Li, Jeffrey Wu, Jianbo Yan, Minghao Bian, Song Sharma, Vedang Sun, Ke Dillmann, Steven Anand, Akshay Lanpouthakoun, Andrew Koopah, Bardia Hu, Changran Guha, Etash Dreiman, Gabriel H. S. Zhu, Jiacheng Krauth, Karl Zhong, Li Muennighoff, Niklas Amanfu, Robert Tan, Shangyin Pimpalgaonkar, Shreyas Aggarwal, Tushar Lin, Xiangning Lan, Xin Zhao, Xuandong Liang, Yiqing Wang, Yuanli Wang, Zilong Zhou, Changzhi Heineman, David Liu, Hange Trivedi, Harsh Yang, John Lin, Junhong Shetty, Manish Yang, Michael Omi, Nabil Raoof, Negin Li, Shanda Zhuo, Terry Yue Lin, Wuwei Dai, Yiwei Wang, Yuxin Chai, Wenhao Zhou, Shang Wahdany, Dariush She, Ziyu Hu, Jiaming Dong, Zhikang Zhu, Yuxuan Cui, Sasha Saiyed, Ahson Kolbeinsson, Arinbjörn Hu, Jesse Rytting, Christopher Michael Marten, Ryan Wang, Yixin Dimakis, Alex Konwinski, Andy Schmidt, Ludwig
author_facet	Merrill, Mike A. Shaw, Alexander G. Carlini, Nicholas Li, Boxuan Raj, Harsh Bercovich, Ivan Shi, Lin Shin, Jeong Yeon Walshe, Thomas Buchanan, E. Kelly Shen, Junhong Ye, Guanghao Lin, Haowei Poulos, Jason Wang, Maoyu Nezhurina, Marianna Jitsev, Jenia Lu, Di Mastromichalakis, Orfeas Menis Xu, Zhiwei Chen, Zizhao Liu, Yue Zhang, Robert Chen, Leon Liangyu Kashyap, Anurag Uslu, Jan-Lucas Li, Jeffrey Wu, Jianbo Yan, Minghao Bian, Song Sharma, Vedang Sun, Ke Dillmann, Steven Anand, Akshay Lanpouthakoun, Andrew Koopah, Bardia Hu, Changran Guha, Etash Dreiman, Gabriel H. S. Zhu, Jiacheng Krauth, Karl Zhong, Li Muennighoff, Niklas Amanfu, Robert Tan, Shangyin Pimpalgaonkar, Shreyas Aggarwal, Tushar Lin, Xiangning Lan, Xin Zhao, Xuandong Liang, Yiqing Wang, Yuanli Wang, Zilong Zhou, Changzhi Heineman, David Liu, Hange Trivedi, Harsh Yang, John Lin, Junhong Shetty, Manish Yang, Michael Omi, Nabil Raoof, Negin Li, Shanda Zhuo, Terry Yue Lin, Wuwei Dai, Yiwei Wang, Yuxin Chai, Wenhao Zhou, Shang Wahdany, Dariush She, Ziyu Hu, Jiaming Dong, Zhikang Zhu, Yuxuan Cui, Sasha Saiyed, Ahson Kolbeinsson, Arinbjörn Hu, Jesse Rytting, Christopher Michael Marten, Ryan Wang, Yixin Dimakis, Alex Konwinski, Andy Schmidt, Ludwig
contents	AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models. To this end, we present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification. We show that frontier models and agents score less than 65\% on the benchmark and conduct an error analysis to identify areas for model and agent improvement. We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/ .
format	Preprint
id	arxiv_https___arxiv_org_abs_2601_11868
institution	arXiv
publishDate	2026
record_format	arxiv
spellingShingle	Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces Merrill, Mike A. Shaw, Alexander G. Carlini, Nicholas Li, Boxuan Raj, Harsh Bercovich, Ivan Shi, Lin Shin, Jeong Yeon Walshe, Thomas Buchanan, E. Kelly Shen, Junhong Ye, Guanghao Lin, Haowei Poulos, Jason Wang, Maoyu Nezhurina, Marianna Jitsev, Jenia Lu, Di Mastromichalakis, Orfeas Menis Xu, Zhiwei Chen, Zizhao Liu, Yue Zhang, Robert Chen, Leon Liangyu Kashyap, Anurag Uslu, Jan-Lucas Li, Jeffrey Wu, Jianbo Yan, Minghao Bian, Song Sharma, Vedang Sun, Ke Dillmann, Steven Anand, Akshay Lanpouthakoun, Andrew Koopah, Bardia Hu, Changran Guha, Etash Dreiman, Gabriel H. S. Zhu, Jiacheng Krauth, Karl Zhong, Li Muennighoff, Niklas Amanfu, Robert Tan, Shangyin Pimpalgaonkar, Shreyas Aggarwal, Tushar Lin, Xiangning Lan, Xin Zhao, Xuandong Liang, Yiqing Wang, Yuanli Wang, Zilong Zhou, Changzhi Heineman, David Liu, Hange Trivedi, Harsh Yang, John Lin, Junhong Shetty, Manish Yang, Michael Omi, Nabil Raoof, Negin Li, Shanda Zhuo, Terry Yue Lin, Wuwei Dai, Yiwei Wang, Yuxin Chai, Wenhao Zhou, Shang Wahdany, Dariush She, Ziyu Hu, Jiaming Dong, Zhikang Zhu, Yuxuan Cui, Sasha Saiyed, Ahson Kolbeinsson, Arinbjörn Hu, Jesse Rytting, Christopher Michael Marten, Ryan Wang, Yixin Dimakis, Alex Konwinski, Andy Schmidt, Ludwig Software Engineering Artificial Intelligence AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models. To this end, we present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification. We show that frontier models and agents score less than 65\% on the benchmark and conduct an error analysis to identify areas for model and agent improvement. We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/ .
title	Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
topic	Software Engineering Artificial Intelligence
url	https://arxiv.org/abs/2601.11868