Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.11621 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929683088539648 |
|---|---|
| author | Bhasin, Vedant Yudin, Matthew Stefanescu, Razvan Izmailov, Rauf |
| author_facet | Bhasin, Vedant Yudin, Matthew Stefanescu, Razvan Izmailov, Rauf |
| contents | Trojan backdoors can be injected into large language models at various stages, including pretraining, fine-tuning, and in-context learning, posing a significant threat to the model's alignment. Due to the nature of causal language modeling, detecting these triggers is challenging given the vast search space. In this study, we propose a multistage framework for detecting Trojan triggers in large language models consisting of token filtration, trigger identification, and trigger verification. We discuss existing trigger identification methods and propose two variants of a black-box trigger inversion method that rely on output logits, utilizing beam search and greedy decoding respectively. We show that the verification stage is critical in the process and propose semantic-preserving prompts and special perturbations to differentiate between actual Trojan triggers and other adversarial strings that display similar characteristics. The evaluation of our approach on the TrojAI and RLHF poisoned model datasets demonstrates promising results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_11621 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Trojan Detection Through Pattern Recognition for Large Language Models Bhasin, Vedant Yudin, Matthew Stefanescu, Razvan Izmailov, Rauf Computation and Language Machine Learning 68T10, 68T20 I.2; I.5 Trojan backdoors can be injected into large language models at various stages, including pretraining, fine-tuning, and in-context learning, posing a significant threat to the model's alignment. Due to the nature of causal language modeling, detecting these triggers is challenging given the vast search space. In this study, we propose a multistage framework for detecting Trojan triggers in large language models consisting of token filtration, trigger identification, and trigger verification. We discuss existing trigger identification methods and propose two variants of a black-box trigger inversion method that rely on output logits, utilizing beam search and greedy decoding respectively. We show that the verification stage is critical in the process and propose semantic-preserving prompts and special perturbations to differentiate between actual Trojan triggers and other adversarial strings that display similar characteristics. The evaluation of our approach on the TrojAI and RLHF poisoned model datasets demonstrates promising results. |
| title | Trojan Detection Through Pattern Recognition for Large Language Models |
| topic | Computation and Language Machine Learning 68T10, 68T20 I.2; I.5 |
| url | https://arxiv.org/abs/2501.11621 |