Cover Image

Modality-Inconsistent Continual Learning of Multimodal Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Pian, Weiguo, Deng, Shijian, Mo, Shentong, Liu, Mingrui, Guo, Yunhui, Tian, Yapeng
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Sound Audio and Speech Processing
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Continual Audio-Visual Sound Separation
by: Pian, Weiguo, et al.
Published: (2024)

MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformers
by: Mahmud, Tanvir, et al.
Published: (2024)

Semantic Grouping Network for Audio Source Separation
by: Mo, Shentong, et al.
Published: (2024)

Text-to-Audio Generation Synchronized with Videos
by: Mo, Shentong, et al.
Published: (2024)

Improving Audio Codec-based Zero-Shot Text-to-Speech Synthesis with Multi-Modal Context and Large Language Model
by: Xue, Jinlong, et al.
Published: (2024)

Closing the Modality Reasoning Gap for Speech Large Language Models
by: Wang, Chaoren, et al.
Published: (2026)

Audio-visual Generalized Zero-shot Learning the Easy Way
by: Mo, Shentong, et al.
Published: (2024)

Fine-Tuning Large Multimodal Models for Automatic Pronunciation Assessment
by: Wang, Ke, et al.
Published: (2025)

CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models
by: Wu, Shangda, et al.
Published: (2024)

Exploring the Potential of Large Multimodal Models as Effective Alternatives for Pronunciation Assessment
by: Wang, Ke, et al.
Published: (2025)

SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Models
by: Peri, Raghuveer, et al.
Published: (2024)

Enhancing Multimodal Emotion Recognition through Multi-Granularity Cross-Modal Alignment
by: Wang, Xuechen, et al.
Published: (2024)

SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation
by: Sun, Chunyu, et al.
Published: (2025)

Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspective
by: Wang, Hankun, et al.
Published: (2024)

Multi-scale Multi-instance Visual Sound Localization and Segmentation
by: Mo, Shentong, et al.
Published: (2024)

What do MLLMs hear? Examining reasoning with text and sound components in Multimodal Large Language Models
by: Çoban, Enis Berk, et al.
Published: (2024)

DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
by: Lu, Ke-Han, et al.
Published: (2025)

SSR: Alignment-Aware Modality Connector for Speech Language Models
by: Tan, Weiting, et al.
Published: (2024)

Self-Powered LLM Modality Expansion for Large Speech-Text Models
by: Yu, Tengfei, et al.
Published: (2024)

Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrieval
by: Zhou, Lifeng, et al.
Published: (2024)

Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Models
by: Sridhar, Arvind Krishna, et al.
Published: (2024)

SALMONN: Towards Generic Hearing Abilities for Large Language Models
by: Tang, Changli, et al.
Published: (2023)

Leveraging Large Language Models for Spontaneous Speech-Based Suicide Risk Detection
by: Gao, Yifan, et al.
Published: (2025)

An End-to-End Speech Summarization Using Large Language Model
by: Shang, Hengchao, et al.
Published: (2024)

StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Model
by: Guo, Shoutao, et al.
Published: (2025)

Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation
by: Wang, Siyin, et al.
Published: (2024)

T-VSL: Text-Guided Visual Sound Source Localization in Mixtures
by: Mahmud, Tanvir, et al.
Published: (2024)

ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Models
by: Li, Bohan, et al.
Published: (2025)

VOX-KRIKRI: Unifying Speech and Language through Continuous Fusion
by: Damianos, Dimitrios, et al.
Published: (2025)

Continual Speech Learning with Fused Speech Features
by: Wang, Guitao, et al.
Published: (2025)

Weight Factorization and Centralization for Continual Learning in Speech Recognition
by: Ugan, Enes Yavuz, et al.
Published: (2025)

Multimodal Input Aids a Bayesian Model of Phonetic Learning
by: Zhi, Sophia, et al.
Published: (2024)

Pay More Attention To Audio: Mitigating Imbalance of Cross-Modal Attention in Large Audio Language Models
by: Wang, Junyu, et al.
Published: (2025)

LESS: Large Language Model Enhanced Semi-Supervised Learning for Speech Foundational Models Using in-the-wild Data
by: Ding, Wen, et al.
Published: (2025)

Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights
by: Yang, Hao, et al.
Published: (2024)

Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model
by: Huang, Ailin, et al.
Published: (2025)

ALAS: Measuring Latent Speech-Text Alignment For Spoken Language Understanding In Multimodal LLMs
by: Mousavi, Pooneh, et al.
Published: (2025)

Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space
by: Ma, Zhengrui, et al.
Published: (2025)

Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows
by: Mo, Shentong, et al.
Published: (2026)

Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
by: Zhang, Shaolei, et al.
Published: (2025)