Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.17764 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914046461083648 |
|---|---|
| author | Zhang, Tianhao Sheng, Zhecheng Lin, Zhexiao Jiang, Chen Kang, Dongyeop |
| author_facet | Zhang, Tianhao Sheng, Zhecheng Lin, Zhexiao Jiang, Chen Kang, Dongyeop |
| contents | Autoregressive generative models play a key role in various language tasks, especially for modeling and evaluating long text sequences. While recent methods leverage stochastic representations to better capture sequence dynamics, encoding both temporal and structural dependencies and utilizing such information for evaluation remains challenging. In this work, we observe that fitting transformer-based model embeddings into a stochastic process yields ordered latent representations from originally unordered model outputs. Building on this insight and prior work, we theoretically introduce a novel likelihood-based evaluation metric BBScoreV2. Empirically, we demonstrate that the stochastic latent space induces a "clustered-to-temporal ordered" mapping of language model representations in high-dimensional space, offering both intuitive and quantitative support for the effectiveness of BBScoreV2. Furthermore, this structure aligns with intrinsic properties of natural language and enhances performance on tasks such as temporal consistency evaluation (e.g., Shuffle tasks) and AI-generated content detection. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17764 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BBScoreV2: Learning Time-Evolution and Latent Alignment from Stochastic Representation Zhang, Tianhao Sheng, Zhecheng Lin, Zhexiao Jiang, Chen Kang, Dongyeop Computation and Language Artificial Intelligence Statistics Theory Autoregressive generative models play a key role in various language tasks, especially for modeling and evaluating long text sequences. While recent methods leverage stochastic representations to better capture sequence dynamics, encoding both temporal and structural dependencies and utilizing such information for evaluation remains challenging. In this work, we observe that fitting transformer-based model embeddings into a stochastic process yields ordered latent representations from originally unordered model outputs. Building on this insight and prior work, we theoretically introduce a novel likelihood-based evaluation metric BBScoreV2. Empirically, we demonstrate that the stochastic latent space induces a "clustered-to-temporal ordered" mapping of language model representations in high-dimensional space, offering both intuitive and quantitative support for the effectiveness of BBScoreV2. Furthermore, this structure aligns with intrinsic properties of natural language and enhances performance on tasks such as temporal consistency evaluation (e.g., Shuffle tasks) and AI-generated content detection. |
| title | BBScoreV2: Learning Time-Evolution and Latent Alignment from Stochastic Representation |
| topic | Computation and Language Artificial Intelligence Statistics Theory |
| url | https://arxiv.org/abs/2405.17764 |