Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.11779 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916540909092864 |
|---|---|
| author | Gross, Jason Agrawal, Rajashree Kwa, Thomas Ong, Euan Yip, Chun Hei Gibson, Alex Noubir, Soufiane Chan, Lawrence |
| author_facet | Gross, Jason Agrawal, Rajashree Kwa, Thomas Ong, Euan Yip, Chun Hei Gibson, Alex Noubir, Soufiane Chan, Lawrence |
| contents | We propose using mechanistic interpretability -- techniques for reverse engineering model weights into human-interpretable algorithms -- to derive and compactly prove formal guarantees on model performance. We prototype this approach by formally proving accuracy lower bounds for a small transformer trained on Max-of-K, validating proof transferability across 151 random seeds and four values of K. We create 102 different computer-assisted proof strategies and assess their length and tightness of bound on each of our models. Using quantitative metrics, we find that shorter proofs seem to require and provide more mechanistic understanding. Moreover, we find that more faithful mechanistic understanding leads to tighter performance bounds. We confirm these connections by qualitatively examining a subset of our proofs. Finally, we identify compounding structureless errors as a key challenge for using mechanistic interpretability to generate compact proofs on model performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11779 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Compact Proofs of Model Performance via Mechanistic Interpretability Gross, Jason Agrawal, Rajashree Kwa, Thomas Ong, Euan Yip, Chun Hei Gibson, Alex Noubir, Soufiane Chan, Lawrence Machine Learning Logic in Computer Science We propose using mechanistic interpretability -- techniques for reverse engineering model weights into human-interpretable algorithms -- to derive and compactly prove formal guarantees on model performance. We prototype this approach by formally proving accuracy lower bounds for a small transformer trained on Max-of-K, validating proof transferability across 151 random seeds and four values of K. We create 102 different computer-assisted proof strategies and assess their length and tightness of bound on each of our models. Using quantitative metrics, we find that shorter proofs seem to require and provide more mechanistic understanding. Moreover, we find that more faithful mechanistic understanding leads to tighter performance bounds. We confirm these connections by qualitatively examining a subset of our proofs. Finally, we identify compounding structureless errors as a key challenge for using mechanistic interpretability to generate compact proofs on model performance. |
| title | Compact Proofs of Model Performance via Mechanistic Interpretability |
| topic | Machine Learning Logic in Computer Science |
| url | https://arxiv.org/abs/2406.11779 |