Where Does Authorship Signal Emerge in Encoder-Based Language Models?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911717635653632 |
|---|---|
| author | Kulumba, Francis Vimont, Guillaume Romary, Laurent Cafiero, Florian |
| author_facet | Kulumba, Francis Vimont, Guillaume Romary, Laurent Cafiero, Florian |
| contents | Authorship attribution models fine-tuned with the same pretrained encoder, data, and loss can differ four-fold in performance depending only on their scoring mechanism. We use mechanistic interpretability tools to explain this gap. Stylistic features such as word length, punctuation density, and function-word frequency are similarly available at every layer in every model we probe, including an off-the-shelf control encoder, suggesting that the gap is not explained by their linear readability. Instead, causal intervention shows that the scorer appears to determine where the encoder consolidates authorship signal. Mean pooling forces consolidation by early to mid layers, while late interaction defers it to later layers. We further derive this difference from the gradient structure of each scorer, and training dynamics reveal distinct learning trajectories that follow from that difference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_19908 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Where Does Authorship Signal Emerge in Encoder-Based Language Models? Kulumba, Francis Vimont, Guillaume Romary, Laurent Cafiero, Florian Computation and Language Authorship attribution models fine-tuned with the same pretrained encoder, data, and loss can differ four-fold in performance depending only on their scoring mechanism. We use mechanistic interpretability tools to explain this gap. Stylistic features such as word length, punctuation density, and function-word frequency are similarly available at every layer in every model we probe, including an off-the-shelf control encoder, suggesting that the gap is not explained by their linear readability. Instead, causal intervention shows that the scorer appears to determine where the encoder consolidates authorship signal. Mean pooling forces consolidation by early to mid layers, while late interaction defers it to later layers. We further derive this difference from the gradient structure of each scorer, and training dynamics reveal distinct learning trajectories that follow from that difference. |
| title | Where Does Authorship Signal Emerge in Encoder-Based Language Models? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.19908 |