LOCOST: State-Space Models for Long Document Abstractive Summarization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916174316437504 |
|---|---|
| author | Bronnec, Florian Le Duong, Song Ravaut, Mathieu Allauzen, Alexandre Chen, Nancy F. Guigue, Vincent Lumbreras, Alberto Soulier, Laure Gallinari, Patrick |
| author_facet | Bronnec, Florian Le Duong, Song Ravaut, Mathieu Allauzen, Alexandre Chen, Nancy F. Guigue, Vincent Lumbreras, Alberto Soulier, Laure Gallinari, Patrick |
| contents | State-space models are a low-complexity alternative to transformers for encoding long sequences and capturing long-term dependencies. We propose LOCOST: an encoder-decoder architecture based on state-space models for conditional text generation with long context inputs. With a computational complexity of $O(L \log L)$, this architecture can handle significantly longer sequences than state-of-the-art models that are based on sparse attention patterns. We evaluate our model on a series of long document abstractive summarization tasks. The model reaches a performance level that is 93-96% comparable to the top-performing sparse transformers of the same size while saving up to 50% memory during training and up to 87% during inference. Additionally, LOCOST effectively handles input texts exceeding 600K tokens at inference time, setting new state-of-the-art results on full-book summarization and opening new perspectives for long input processing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_17919 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LOCOST: State-Space Models for Long Document Abstractive Summarization Bronnec, Florian Le Duong, Song Ravaut, Mathieu Allauzen, Alexandre Chen, Nancy F. Guigue, Vincent Lumbreras, Alberto Soulier, Laure Gallinari, Patrick Computation and Language Machine Learning State-space models are a low-complexity alternative to transformers for encoding long sequences and capturing long-term dependencies. We propose LOCOST: an encoder-decoder architecture based on state-space models for conditional text generation with long context inputs. With a computational complexity of $O(L \log L)$, this architecture can handle significantly longer sequences than state-of-the-art models that are based on sparse attention patterns. We evaluate our model on a series of long document abstractive summarization tasks. The model reaches a performance level that is 93-96% comparable to the top-performing sparse transformers of the same size while saving up to 50% memory during training and up to 87% during inference. Additionally, LOCOST effectively handles input texts exceeding 600K tokens at inference time, setting new state-of-the-art results on full-book summarization and opening new perspectives for long input processing. |
| title | LOCOST: State-Space Models for Long Document Abstractive Summarization |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2401.17919 |