Mining Word Boundaries from Speech-Text Parallel Data for Cross-domain Chinese Word Segmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909425366728704 |
|---|---|
| author | Wang, Xuebin Zhang, Lei Li, Zhenghua Zhou, Shilin Gong, Chen Hou, Yang |
| author_facet | Wang, Xuebin Zhang, Lei Li, Zhenghua Zhou, Shilin Gong, Chen Hou, Yang |
| contents | Inspired by early research on exploring naturally annotated data for Chinese Word Segmentation (CWS), and also by recent research on integration of speech and text processing, this work for the first time proposes to explicitly mine word boundaries from speech-text parallel data. We employ the Montreal Forced Aligner (MFA) toolkit to perform character-level alignment on speech-text data, giving pauses as candidate word boundaries. Based on detailed analysis of collected pauses, we propose an effective probability-based strategy for filtering unreliable word boundaries. To more effectively utilize word boundaries as extra training data, we also propose a robust complete-then-train (CTT) strategy. We conduct cross-domain CWS experiments on two target domains, i.e., ZX and AISHELL2. We have annotated about 1,000 sentences as the evaluation data of AISHELL2. Experiments demonstrate the effectiveness of our proposed approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_09045 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mining Word Boundaries from Speech-Text Parallel Data for Cross-domain Chinese Word Segmentation Wang, Xuebin Zhang, Lei Li, Zhenghua Zhou, Shilin Gong, Chen Hou, Yang Computation and Language Inspired by early research on exploring naturally annotated data for Chinese Word Segmentation (CWS), and also by recent research on integration of speech and text processing, this work for the first time proposes to explicitly mine word boundaries from speech-text parallel data. We employ the Montreal Forced Aligner (MFA) toolkit to perform character-level alignment on speech-text data, giving pauses as candidate word boundaries. Based on detailed analysis of collected pauses, we propose an effective probability-based strategy for filtering unreliable word boundaries. To more effectively utilize word boundaries as extra training data, we also propose a robust complete-then-train (CTT) strategy. We conduct cross-domain CWS experiments on two target domains, i.e., ZX and AISHELL2. We have annotated about 1,000 sentences as the evaluation data of AISHELL2. Experiments demonstrate the effectiveness of our proposed approach. |
| title | Mining Word Boundaries from Speech-Text Parallel Data for Cross-domain Chinese Word Segmentation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.09045 |