Intermittent Semi-Working Mask: A New Masking Paradigm for LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910024036515840 |
|---|---|
| author | Hu, HaoYuan Lu, Mingcong Luo, Di Wu, XinYa Zhu, Jiangcai Yin, Taoye Li, Zheng Wang, Hao Zhang, Shusheng Zhang, KeZun Shao, KaiLai Chen, Chao Wang, Feng |
| author_facet | Hu, HaoYuan Lu, Mingcong Luo, Di Wu, XinYa Zhu, Jiangcai Yin, Taoye Li, Zheng Wang, Hao Zhang, Shusheng Zhang, KeZun Shao, KaiLai Chen, Chao Wang, Feng |
| contents | Multi-turn dialogues and context-intensive tasks challenge Large Language Models (LLMs) to integrate long histories without sacrificing generation quality. Although prefix LLMs can better exploit historical context via bidirectional attention on prefix tokens, they are rarely used in practice because multi-turn training requires many duplicated triplets, and its bidirectional prefix prevents KV-cache reuse at inference time, driving up high cost and latency. To retain the contextual understanding of prefix mask while preserving the inference-time efficiency of causal mask, we introduce Intermittent Semi-working Mask (ISM), a masking scheme that injects sparse bidirectional attention into the causal backbone. ISM alternates bidirectional attention over query segments with unidirectional attention over answer segments, enabling the synthesis of in-context while preserving global causality. This design eliminates triplet expansion during training and maintains KV-cache reuse during inference, yielding latency comparable to standard causal LLMs. ISM is architecture-agnostic and parameter-free, adding only minimal latency. Across extensive evaluations, ISM outperforms causal baselines not only on multi-turn dialogue, but also on context-intensive tasks like mathematical reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_00539 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Intermittent Semi-Working Mask: A New Masking Paradigm for LLMs Hu, HaoYuan Lu, Mingcong Luo, Di Wu, XinYa Zhu, Jiangcai Yin, Taoye Li, Zheng Wang, Hao Zhang, Shusheng Zhang, KeZun Shao, KaiLai Chen, Chao Wang, Feng Computation and Language Artificial Intelligence Multi-turn dialogues and context-intensive tasks challenge Large Language Models (LLMs) to integrate long histories without sacrificing generation quality. Although prefix LLMs can better exploit historical context via bidirectional attention on prefix tokens, they are rarely used in practice because multi-turn training requires many duplicated triplets, and its bidirectional prefix prevents KV-cache reuse at inference time, driving up high cost and latency. To retain the contextual understanding of prefix mask while preserving the inference-time efficiency of causal mask, we introduce Intermittent Semi-working Mask (ISM), a masking scheme that injects sparse bidirectional attention into the causal backbone. ISM alternates bidirectional attention over query segments with unidirectional attention over answer segments, enabling the synthesis of in-context while preserving global causality. This design eliminates triplet expansion during training and maintains KV-cache reuse during inference, yielding latency comparable to standard causal LLMs. ISM is architecture-agnostic and parameter-free, adding only minimal latency. Across extensive evaluations, ISM outperforms causal baselines not only on multi-turn dialogue, but also on context-intensive tasks like mathematical reasoning. |
| title | Intermittent Semi-Working Mask: A New Masking Paradigm for LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2408.00539 |