Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.25884 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913069946372096 |
|---|---|
| author | Cao, Shuxiang Zhang, Zijian Agarwal, Abhishek Bratrud, Grace Beysengulov, Niyaz R. Cole, Daniel C. Frieiro, Alejandro Gómez Glen, Elena O. Hsu, Hao Huang, Gang Jow, Raymond Shaji, Greshma Lubowe, Tom Zhu, Ligeng Calderón, Luis Mantilla Pancotti, Nicola Pendleton, Joel Severin, Brandon Staub, Charles Etienne Sussman, Sara Vepsäläinen, Antti Vora, Neel Rajeshbhai Xu, Yilun Bernales, Varinia Bowring, Daniel Kyoseva, Elica Rungger, Ivan Semeghini, Giulia Stanwyck, Sam Costa, Timothy Aspuru-Guzik, Alán Svore, Krysta |
| author_facet | Cao, Shuxiang Zhang, Zijian Agarwal, Abhishek Bratrud, Grace Beysengulov, Niyaz R. Cole, Daniel C. Frieiro, Alejandro Gómez Glen, Elena O. Hsu, Hao Huang, Gang Jow, Raymond Shaji, Greshma Lubowe, Tom Zhu, Ligeng Calderón, Luis Mantilla Pancotti, Nicola Pendleton, Joel Severin, Brandon Staub, Charles Etienne Sussman, Sara Vepsäläinen, Antti Vora, Neel Rajeshbhai Xu, Yilun Bernales, Varinia Bowring, Daniel Kyoseva, Elica Rungger, Ivan Semeghini, Giulia Stanwyck, Sam Costa, Timothy Aspuru-Guzik, Alán Svore, Krysta |
| contents | Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-language models (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_25884 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding Cao, Shuxiang Zhang, Zijian Agarwal, Abhishek Bratrud, Grace Beysengulov, Niyaz R. Cole, Daniel C. Frieiro, Alejandro Gómez Glen, Elena O. Hsu, Hao Huang, Gang Jow, Raymond Shaji, Greshma Lubowe, Tom Zhu, Ligeng Calderón, Luis Mantilla Pancotti, Nicola Pendleton, Joel Severin, Brandon Staub, Charles Etienne Sussman, Sara Vepsäläinen, Antti Vora, Neel Rajeshbhai Xu, Yilun Bernales, Varinia Bowring, Daniel Kyoseva, Elica Rungger, Ivan Semeghini, Giulia Stanwyck, Sam Costa, Timothy Aspuru-Guzik, Alán Svore, Krysta Quantum Physics Computer Vision and Pattern Recognition Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-language models (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score. |
| title | QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding |
| topic | Quantum Physics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2604.25884 |