MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910968384061440 |
|---|---|
| author | Zhao, Jihao Ji, Zhiyuan Fan, Zhaoxin Wang, Hanyu Niu, Simin Tang, Bo Xiong, Feiyu Li, Zhiyu |
| author_facet | Zhao, Jihao Ji, Zhiyuan Fan, Zhaoxin Wang, Hanyu Niu, Simin Tang, Bo Xiong, Feiyu Li, Zhiyu |
| contents | Retrieval-Augmented Generation (RAG), while serving as a viable complement to large language models (LLMs), often overlooks the crucial aspect of text chunking within its pipeline. This paper initially introduces a dual-metric evaluation method, comprising Boundary Clarity and Chunk Stickiness, to enable the direct quantification of chunking quality. Leveraging this assessment method, we highlight the inherent limitations of traditional and semantic chunking in handling complex contextual nuances, thereby substantiating the necessity of integrating LLMs into chunking process. To address the inherent trade-off between computational efficiency and chunking precision in LLM-based approaches, we devise the granularity-aware Mixture-of-Chunkers (MoC) framework, which consists of a three-stage processing mechanism. Notably, our objective is to guide the chunker towards generating a structured list of chunking regular expressions, which are subsequently employed to extract chunks from the original text. Extensive experiments demonstrate that both our proposed metrics and the MoC framework effectively settle challenges of the chunking task, revealing the chunking kernel while enhancing the performance of the RAG system. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_09600 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System Zhao, Jihao Ji, Zhiyuan Fan, Zhaoxin Wang, Hanyu Niu, Simin Tang, Bo Xiong, Feiyu Li, Zhiyu Computation and Language Retrieval-Augmented Generation (RAG), while serving as a viable complement to large language models (LLMs), often overlooks the crucial aspect of text chunking within its pipeline. This paper initially introduces a dual-metric evaluation method, comprising Boundary Clarity and Chunk Stickiness, to enable the direct quantification of chunking quality. Leveraging this assessment method, we highlight the inherent limitations of traditional and semantic chunking in handling complex contextual nuances, thereby substantiating the necessity of integrating LLMs into chunking process. To address the inherent trade-off between computational efficiency and chunking precision in LLM-based approaches, we devise the granularity-aware Mixture-of-Chunkers (MoC) framework, which consists of a three-stage processing mechanism. Notably, our objective is to guide the chunker towards generating a structured list of chunking regular expressions, which are subsequently employed to extract chunks from the original text. Extensive experiments demonstrate that both our proposed metrics and the MoC framework effectively settle challenges of the chunking task, revealing the chunking kernel while enhancing the performance of the RAG system. |
| title | MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.09600 |