Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.19790 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918460961849344 |
|---|---|
| author | Wang, Yifei Li, Tianlin Zhang, Xiaohan Zhang, Xiaoyu Ma, Wei Cheng, Mingfei Pan, Li |
| author_facet | Wang, Yifei Li, Tianlin Zhang, Xiaohan Zhang, Xiaoyu Ma, Wei Cheng, Mingfei Pan, Li |
| contents | Large language models (LLMs) are increasingly deployed under diverse numerical precision configurations, including standard floating-point formats (e.g., bfloat16 and float16) and quantized integer formats (e.g., int16 and int8), to meet efficiency and resource constraints. However, minor inconsistencies between LLMs of different precisions are difficult to detect and are often overlooked by existing evaluation methods. In this paper, we present PrecisionDiff, an automated differential testing framework for systematically detecting precision-induced behavioral disagreements in LLMs. PrecisionDiff generates precision-sensitive test inputs and performs cross-precision comparative analysis to uncover subtle divergences that remain hidden under conventional testing strategies. To demonstrate its practical significance, we instantiate PrecisionDiff on the alignment verification task, where precision-induced disagreements manifest as jailbreak divergence-inputs that are rejected under one precision may produce harmful responses under another. Experimental results show that such behavioral disagreements are widespread across multiple open-source aligned LLMs and precision settings, and that PrecisionDiff significantly outperforms vanilla testing methods in detecting these issues. Our work enables automated precision-sensitive test generation, facilitating effective pre-deployment evaluation and improving precision robustness during training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_19790 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements Wang, Yifei Li, Tianlin Zhang, Xiaohan Zhang, Xiaoyu Ma, Wei Cheng, Mingfei Pan, Li Artificial Intelligence Machine Learning I.2.7; K.6.5 Large language models (LLMs) are increasingly deployed under diverse numerical precision configurations, including standard floating-point formats (e.g., bfloat16 and float16) and quantized integer formats (e.g., int16 and int8), to meet efficiency and resource constraints. However, minor inconsistencies between LLMs of different precisions are difficult to detect and are often overlooked by existing evaluation methods. In this paper, we present PrecisionDiff, an automated differential testing framework for systematically detecting precision-induced behavioral disagreements in LLMs. PrecisionDiff generates precision-sensitive test inputs and performs cross-precision comparative analysis to uncover subtle divergences that remain hidden under conventional testing strategies. To demonstrate its practical significance, we instantiate PrecisionDiff on the alignment verification task, where precision-induced disagreements manifest as jailbreak divergence-inputs that are rejected under one precision may produce harmful responses under another. Experimental results show that such behavioral disagreements are widespread across multiple open-source aligned LLMs and precision settings, and that PrecisionDiff significantly outperforms vanilla testing methods in detecting these issues. Our work enables automated precision-sensitive test generation, facilitating effective pre-deployment evaluation and improving precision robustness during training. |
| title | Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements |
| topic | Artificial Intelligence Machine Learning I.2.7; K.6.5 |
| url | https://arxiv.org/abs/2604.19790 |