Seed-Induced Uniqueness in Transformer Models: Subspace Alignment Governs Subliminal Transfer
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915800215977984 |
|---|---|
| author | Okatan, Ayşe Selin Akbaş, Mustafa İlhan Kandel, Laxima Niure Peköz, Berker |
| author_facet | Okatan, Ayşe Selin Akbaş, Mustafa İlhan Kandel, Laxima Niure Peköz, Berker |
| contents | We analyze subliminal transfer in Transformer models, where a teacher embeds hidden traits that can be linearly decoded by a student without degrading main-task performance. Prior work often attributes transferability to global representational similarity, typically quantified with Centered Kernel Alignment (CKA). Using synthetic corpora with disentangled public and private labels, we distill students under matched and independent random initializations. We find that transfer strength hinges on alignment within a trait-discriminative subspace: same-seed students inherit this alignment and show higher leakage {τ\approx} 0.24, whereas different-seed students -- despite global CKA > 0.9 -- exhibit substantially reduced excess accuracy {τ\approx} 0.12 - 0.13. We formalize this with subspace-level CKA diagnostic and residualized probes, showing that leakage tracks alignment within the trait-discriminative subspace rather than global representational similarity. Security controls (projection penalty, adversarial reversal, right-for-the-wrong-reasons regularization) reduce leakage in same-base models without impairing public-task fidelity. These results establish seed-induced uniqueness as a resilience property and argue for subspace-aware diagnostics for secure multi-model deployments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_01023 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Seed-Induced Uniqueness in Transformer Models: Subspace Alignment Governs Subliminal Transfer Okatan, Ayşe Selin Akbaş, Mustafa İlhan Kandel, Laxima Niure Peköz, Berker Signal Processing Artificial Intelligence Cryptography and Security Machine Learning 68T07, 68P25, 94A60, 68Q32, 68Q87, 94A17 I.2.6; C.2.0; D.4.6; E.3; I.5.1; K.6.5 We analyze subliminal transfer in Transformer models, where a teacher embeds hidden traits that can be linearly decoded by a student without degrading main-task performance. Prior work often attributes transferability to global representational similarity, typically quantified with Centered Kernel Alignment (CKA). Using synthetic corpora with disentangled public and private labels, we distill students under matched and independent random initializations. We find that transfer strength hinges on alignment within a trait-discriminative subspace: same-seed students inherit this alignment and show higher leakage {τ\approx} 0.24, whereas different-seed students -- despite global CKA > 0.9 -- exhibit substantially reduced excess accuracy {τ\approx} 0.12 - 0.13. We formalize this with subspace-level CKA diagnostic and residualized probes, showing that leakage tracks alignment within the trait-discriminative subspace rather than global representational similarity. Security controls (projection penalty, adversarial reversal, right-for-the-wrong-reasons regularization) reduce leakage in same-base models without impairing public-task fidelity. These results establish seed-induced uniqueness as a resilience property and argue for subspace-aware diagnostics for secure multi-model deployments. |
| title | Seed-Induced Uniqueness in Transformer Models: Subspace Alignment Governs Subliminal Transfer |
| topic | Signal Processing Artificial Intelligence Cryptography and Security Machine Learning 68T07, 68P25, 94A60, 68Q32, 68Q87, 94A17 I.2.6; C.2.0; D.4.6; E.3; I.5.1; K.6.5 |
| url | https://arxiv.org/abs/2511.01023 |