Logit-KL Flow Matching: Non-Autoregressive Text Generation via Sampling-Hybrid Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914527851839488 |
|---|---|
| author | Sevriugov, Egor Dragunov, Nikita Razzhigaev, Anton Kuznetsov, Andrey Oseledets, Ivan |
| author_facet | Sevriugov, Egor Dragunov, Nikita Razzhigaev, Anton Kuznetsov, Andrey Oseledets, Ivan |
| contents | Non-autoregressive (NAR) language models offer notable efficiency in text generation by circumventing the sequential bottleneck of autoregressive decoding. However, accurately modeling dependencies in discrete sequences remains challenging in this paradigm. In this work, we advance the field of NAR generation by applying conditional flow matching (CFM) methods grounded in geometrically principled interpolation, specifically leveraging Kullback-Leibler (KL) divergence geodesics, which correspond to linear interpolation in logit space. We rigorously establish that maximizing conditional likelihood in this setting precisely recovers the flow matching velocity field, supplying the theoretical justification for this approach in sequence modeling. To address practical performance gaps of basic inference, we propose a novel empirical sampling strategy that iteratively denoises and re-noises, along with a hybrid scheme that integrates our sampling method with basic procedure. Across unconditional and conditional text and code infilling, the approach improves perplexity and downstream metrics over prior NAR baselines under matched settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16821 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Logit-KL Flow Matching: Non-Autoregressive Text Generation via Sampling-Hybrid Inference Sevriugov, Egor Dragunov, Nikita Razzhigaev, Anton Kuznetsov, Andrey Oseledets, Ivan Computation and Language Machine Learning Non-autoregressive (NAR) language models offer notable efficiency in text generation by circumventing the sequential bottleneck of autoregressive decoding. However, accurately modeling dependencies in discrete sequences remains challenging in this paradigm. In this work, we advance the field of NAR generation by applying conditional flow matching (CFM) methods grounded in geometrically principled interpolation, specifically leveraging Kullback-Leibler (KL) divergence geodesics, which correspond to linear interpolation in logit space. We rigorously establish that maximizing conditional likelihood in this setting precisely recovers the flow matching velocity field, supplying the theoretical justification for this approach in sequence modeling. To address practical performance gaps of basic inference, we propose a novel empirical sampling strategy that iteratively denoises and re-noises, along with a hybrid scheme that integrates our sampling method with basic procedure. Across unconditional and conditional text and code infilling, the approach improves perplexity and downstream metrics over prior NAR baselines under matched settings. |
| title | Logit-KL Flow Matching: Non-Autoregressive Text Generation via Sampling-Hybrid Inference |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2411.16821 |