Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.10407 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916000727826432 |
|---|---|
| author | Nie, Wenhua Zhu, ZiCheng Wu, Jianan Luo, Binhan Zheng, Haoran Jang, Jyh-Shing Roger |
| author_facet | Nie, Wenhua Zhu, ZiCheng Wu, Jianan Luo, Binhan Zheng, Haoran Jang, Jyh-Shing Roger |
| contents | Modern LLM APIs often reveal only top-$K$ logit scores and censor the remaining vocabulary. We study the per-position distribution-recovery limits of this access model. For censoring threshold $τ$, the compatible teacher distributions form an identified set whose total-variation diameter is exactly $U_K=(V-K)\exp(τ)/(Z_A+(V-K)\exp(τ))$, where $Z_A$ is the observed partition function. For KL recovery, we give a computable binary-endpoint lower bound and an asymptotically matching small-ambiguity upper bound, with an extension to reference-aware attackers. Experiments on a Qwen3 math-reasoning teacher reveal a layered extraction hierarchy: on-task top-$K$ distillation recovers 12% of private capability, full-logit distillation recovers 56% despite 99% KL closure, and generation-based extraction recovers 96%. Top-$K$ censoring therefore limits per-position distribution recovery but does not by itself prevent capability extraction, separating fidelity from transfer in prompt-only logit distillation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_10407 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Identified-Set Geometry of Distributional Model Extraction under Top-$K$ Censored API Access Nie, Wenhua Zhu, ZiCheng Wu, Jianan Luo, Binhan Zheng, Haoran Jang, Jyh-Shing Roger Machine Learning Modern LLM APIs often reveal only top-$K$ logit scores and censor the remaining vocabulary. We study the per-position distribution-recovery limits of this access model. For censoring threshold $τ$, the compatible teacher distributions form an identified set whose total-variation diameter is exactly $U_K=(V-K)\exp(τ)/(Z_A+(V-K)\exp(τ))$, where $Z_A$ is the observed partition function. For KL recovery, we give a computable binary-endpoint lower bound and an asymptotically matching small-ambiguity upper bound, with an extension to reference-aware attackers. Experiments on a Qwen3 math-reasoning teacher reveal a layered extraction hierarchy: on-task top-$K$ distillation recovers 12% of private capability, full-logit distillation recovers 56% despite 99% KL closure, and generation-based extraction recovers 96%. Top-$K$ censoring therefore limits per-position distribution recovery but does not by itself prevent capability extraction, separating fidelity from transfer in prompt-only logit distillation. |
| title | Identified-Set Geometry of Distributional Model Extraction under Top-$K$ Censored API Access |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.10407 |