Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913017080315904 |
|---|---|
| author | Zhu, Xiaoyuan Ye, Yaowen Qiu, Tianyi Zhu, Hanlin Tan, Sijun Mannan, Ajraf Michala, Jonathan Popa, Raluca Ada Neiswanger, Willie |
| author_facet | Zhu, Xiaoyuan Ye, Yaowen Qiu, Tianyi Zhu, Hanlin Tan, Sijun Mannan, Ajraf Michala, Jonathan Popa, Raluca Ada Neiswanger, Willie |
| contents | As API access becomes a primary interface to large language models (LLMs), users often interact with black-box systems that offer little transparency into the deployed model. To reduce costs or maliciously alter model behaviors, API providers may discreetly serve quantized or fine-tuned variants, which can degrade performance and compromise safety. Detecting such substitutions is difficult, as users lack access to model weights and, in most cases, even output logits. To tackle this problem, we propose a rank-based uniformity test that can verify the behavioral equality of a black-box LLM to a locally deployed authentic model. Our method is accurate, query-efficient, and avoids detectable query patterns, making it robust to adversarial providers that reroute or mix responses upon the detection of testing attempts. We evaluate the approach across diverse threat scenarios, including quantization, harmful fine-tuning, jailbreak prompts, and full model substitution, showing that it consistently achieves superior statistical power over prior methods under constrained query budgets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_06975 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test Zhu, Xiaoyuan Ye, Yaowen Qiu, Tianyi Zhu, Hanlin Tan, Sijun Mannan, Ajraf Michala, Jonathan Popa, Raluca Ada Neiswanger, Willie Cryptography and Security Artificial Intelligence Computation and Language As API access becomes a primary interface to large language models (LLMs), users often interact with black-box systems that offer little transparency into the deployed model. To reduce costs or maliciously alter model behaviors, API providers may discreetly serve quantized or fine-tuned variants, which can degrade performance and compromise safety. Detecting such substitutions is difficult, as users lack access to model weights and, in most cases, even output logits. To tackle this problem, we propose a rank-based uniformity test that can verify the behavioral equality of a black-box LLM to a locally deployed authentic model. Our method is accurate, query-efficient, and avoids detectable query patterns, making it robust to adversarial providers that reroute or mix responses upon the detection of testing attempts. We evaluate the approach across diverse threat scenarios, including quantization, harmful fine-tuning, jailbreak prompts, and full model substitution, showing that it consistently achieves superior statistical power over prior methods under constrained query budgets. |
| title | Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test |
| topic | Cryptography and Security Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2506.06975 |