The Conformer Encoder May Reverse the Time Dimension
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910785043693568 |
|---|---|
| author | Schmitt, Robin Zeyer, Albert Zeineldeen, Mohammad Schlüter, Ralf Ney, Hermann |
| author_facet | Schmitt, Robin Zeyer, Albert Zeineldeen, Mohammad Schlüter, Ralf Ney, Hermann |
| contents | We sometimes observe monotonically decreasing cross-attention weights in our Conformer-based global attention-based encoder-decoder (AED) models, Further investigation shows that the Conformer encoder reverses the sequence in the time dimension. We analyze the initial behavior of the decoder cross-attention mechanism and find that it encourages the Conformer encoder self-attention to build a connection between the initial frames and all other informative frames. Furthermore, we show that, at some point in training, the self-attention module of the Conformer starts dominating the output over the preceding feed-forward module, which then only allows the reversed information to pass through. We propose methods and ideas of how this flipping can be avoided and investigate a novel method to obtain label-frame-position alignments by using the gradients of the label log probabilities w.r.t. the encoder input frames. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_00680 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Conformer Encoder May Reverse the Time Dimension Schmitt, Robin Zeyer, Albert Zeineldeen, Mohammad Schlüter, Ralf Ney, Hermann Audio and Speech Processing Sound Machine Learning We sometimes observe monotonically decreasing cross-attention weights in our Conformer-based global attention-based encoder-decoder (AED) models, Further investigation shows that the Conformer encoder reverses the sequence in the time dimension. We analyze the initial behavior of the decoder cross-attention mechanism and find that it encourages the Conformer encoder self-attention to build a connection between the initial frames and all other informative frames. Furthermore, we show that, at some point in training, the self-attention module of the Conformer starts dominating the output over the preceding feed-forward module, which then only allows the reversed information to pass through. We propose methods and ideas of how this flipping can be avoided and investigate a novel method to obtain label-frame-position alignments by using the gradients of the label log probabilities w.r.t. the encoder input frames. |
| title | The Conformer Encoder May Reverse the Time Dimension |
| topic | Audio and Speech Processing Sound Machine Learning |
| url | https://arxiv.org/abs/2410.00680 |