Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.21014 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918401313603584 |
|---|---|
| author | Draye, Florent Harrasse, Abir Palit, Vedant Wu, Tung-Yu Liu, Jiarui Pandey, Punya Syon Wu, Roderick Zhang, Terry Jingchen Jin, Zhijing Schölkopf, Bernhard |
| author_facet | Draye, Florent Harrasse, Abir Palit, Vedant Wu, Tung-Yu Liu, Jiarui Pandey, Punya Syon Wu, Roderick Zhang, Terry Jingchen Jin, Zhijing Schölkopf, Bernhard |
| contents | Mechanistic interpretability seeks to understand how Large Language Models (LLMs) represent and process information. Recent approaches based on dictionary learning and transcoders enable representing model computation in terms of sparse, interpretable features and their interactions, giving rise to feature attribution graphs. However, these graphs are often large and redundant, limiting their interpretability in practice. Cross-Layer Transcoders (CLTs) address this issue by sharing features across layers while preserving layer-specific decoding, yielding more compact representations, but remain difficult to train and analyze at scale. We introduce an open-source library for end-to-end training and interpretability of CLTs. Our framework integrates scalable distributed training with model sharding and compressed activation caching, a unified automated interpretability pipeline for feature analysis and explanation, attribution graph computation using Circuit-Tracer, and a flexible visualization interface. This provides a practical and unified solution for scaling CLT-based mechanistic interpretability. Our code is available at: https://github.com/LLM-Interp/CLT-Forge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_21014 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphs Draye, Florent Harrasse, Abir Palit, Vedant Wu, Tung-Yu Liu, Jiarui Pandey, Punya Syon Wu, Roderick Zhang, Terry Jingchen Jin, Zhijing Schölkopf, Bernhard Machine Learning Computation and Language Mechanistic interpretability seeks to understand how Large Language Models (LLMs) represent and process information. Recent approaches based on dictionary learning and transcoders enable representing model computation in terms of sparse, interpretable features and their interactions, giving rise to feature attribution graphs. However, these graphs are often large and redundant, limiting their interpretability in practice. Cross-Layer Transcoders (CLTs) address this issue by sharing features across layers while preserving layer-specific decoding, yielding more compact representations, but remain difficult to train and analyze at scale. We introduce an open-source library for end-to-end training and interpretability of CLTs. Our framework integrates scalable distributed training with model sharding and compressed activation caching, a unified automated interpretability pipeline for feature analysis and explanation, attribution graph computation using Circuit-Tracer, and a flexible visualization interface. This provides a practical and unified solution for scaling CLT-based mechanistic interpretability. Our code is available at: https://github.com/LLM-Interp/CLT-Forge. |
| title | CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphs |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2603.21014 |