wav2graph: A Framework for Supervised Learning Knowledge Graph from Speech
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914905592954880 |
|---|---|
| author | Le-Duc, Khai Dang, Quy-Anh Pham, Tan-Hanh Hy, Truong-Son |
| author_facet | Le-Duc, Khai Dang, Quy-Anh Pham, Tan-Hanh Hy, Truong-Son |
| contents | Knowledge graphs (KGs) enhance the performance of large language models (LLMs) and search engines by providing structured, interconnected data that improves reasoning and context-awareness. However, KGs only focus on text data, thereby neglecting other modalities such as speech. In this work, we introduce wav2graph, the first framework for supervised learning knowledge graph from speech data. Our pipeline are straightforward: (1) constructing a KG based on transcribed spoken utterances and a named entity database, (2) converting KG into embedding vectors, and (3) training graph neural networks (GNNs) for node classification and link prediction tasks. Through extensive experiments conducted in inductive and transductive learning contexts using state-of-the-art GNN models, we provide baseline results and error analysis for node classification and link prediction tasks on human transcripts and automatic speech recognition (ASR) transcripts, including evaluations using both encoder-based and decoder-based node embeddings, as well as monolingual and multilingual acoustic pre-trained models. All related code, data, and models are published online. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_04174 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | wav2graph: A Framework for Supervised Learning Knowledge Graph from Speech Le-Duc, Khai Dang, Quy-Anh Pham, Tan-Hanh Hy, Truong-Son Computation and Language Artificial Intelligence Information Retrieval Machine Learning Sound Audio and Speech Processing Knowledge graphs (KGs) enhance the performance of large language models (LLMs) and search engines by providing structured, interconnected data that improves reasoning and context-awareness. However, KGs only focus on text data, thereby neglecting other modalities such as speech. In this work, we introduce wav2graph, the first framework for supervised learning knowledge graph from speech data. Our pipeline are straightforward: (1) constructing a KG based on transcribed spoken utterances and a named entity database, (2) converting KG into embedding vectors, and (3) training graph neural networks (GNNs) for node classification and link prediction tasks. Through extensive experiments conducted in inductive and transductive learning contexts using state-of-the-art GNN models, we provide baseline results and error analysis for node classification and link prediction tasks on human transcripts and automatic speech recognition (ASR) transcripts, including evaluations using both encoder-based and decoder-based node embeddings, as well as monolingual and multilingual acoustic pre-trained models. All related code, data, and models are published online. |
| title | wav2graph: A Framework for Supervised Learning Knowledge Graph from Speech |
| topic | Computation and Language Artificial Intelligence Information Retrieval Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2408.04174 |