Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.15166 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917729758347264 |
|---|---|
| author | de Bos, Niels uit Garriga-Alonso, Adrià |
| author_facet | de Bos, Niels uit Garriga-Alonso, Adrià |
| contents | Circuits are supposed to accurately describe how a neural network performs a specific task, but do they really? We evaluate three circuits found in the literature (IOI, greater-than, and docstring) in an adversarial manner, considering inputs where the circuit's behavior maximally diverges from the full model. Concretely, we measure the KL divergence between the full model's output and the circuit's output, calculated through resample ablation, and we analyze the worst-performing inputs. Our results show that the circuits for the IOI and docstring tasks fail to behave similarly to the full model even on completely benign inputs from the original task, indicating that more robust circuits are needed for safety-critical applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_15166 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Adversarial Circuit Evaluation de Bos, Niels uit Garriga-Alonso, Adrià Machine Learning Circuits are supposed to accurately describe how a neural network performs a specific task, but do they really? We evaluate three circuits found in the literature (IOI, greater-than, and docstring) in an adversarial manner, considering inputs where the circuit's behavior maximally diverges from the full model. Concretely, we measure the KL divergence between the full model's output and the circuit's output, calculated through resample ablation, and we analyze the worst-performing inputs. Our results show that the circuits for the IOI and docstring tasks fail to behave similarly to the full model even on completely benign inputs from the original task, indicating that more robust circuits are needed for safety-critical applications. |
| title | Adversarial Circuit Evaluation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2407.15166 |