Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2404.15420 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912099026862080 |
|---|---|
| author | Monteiro, João Marcotte, Étienne Noël, Pierre-André Zantedeschi, Valentina Vázquez, David Chapados, Nicolas Pal, Christopher Taslakian, Perouz |
| author_facet | Monteiro, João Marcotte, Étienne Noël, Pierre-André Zantedeschi, Valentina Vázquez, David Chapados, Nicolas Pal, Christopher Taslakian, Perouz |
| contents | In-context learning (ICL) approaches typically leverage prompting to condition decoder-only language model generation on reference information. Just-in-time processing of a context is inefficient due to the quadratic cost of self-attention operations, and caching is desirable. However, caching transformer states can easily require almost as much space as the model parameters. When the right context isn't known in advance, caching ICL can be challenging. This work addresses these limitations by introducing models that, inspired by the encoder-decoder architecture, use cross-attention to condition generation on reference text without the prompt. More precisely, we leverage pre-trained decoder-only models and only train a small number of added layers. We use Question-Answering (QA) as a testbed to evaluate the ability of our models to perform conditional generation and observe that they outperform ICL, are comparable to fine-tuned prompted LLMs, and drastically reduce the space footprint relative to standard KV caching by two orders of magnitude. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_15420 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference Monteiro, João Marcotte, Étienne Noël, Pierre-André Zantedeschi, Valentina Vázquez, David Chapados, Nicolas Pal, Christopher Taslakian, Perouz Computation and Language Artificial Intelligence In-context learning (ICL) approaches typically leverage prompting to condition decoder-only language model generation on reference information. Just-in-time processing of a context is inefficient due to the quadratic cost of self-attention operations, and caching is desirable. However, caching transformer states can easily require almost as much space as the model parameters. When the right context isn't known in advance, caching ICL can be challenging. This work addresses these limitations by introducing models that, inspired by the encoder-decoder architecture, use cross-attention to condition generation on reference text without the prompt. More precisely, we leverage pre-trained decoder-only models and only train a small number of added layers. We use Question-Answering (QA) as a testbed to evaluate the ability of our models to perform conditional generation and observe that they outperform ICL, are comparable to fine-tuned prompted LLMs, and drastically reduce the space footprint relative to standard KV caching by two orders of magnitude. |
| title | XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2404.15420 |