Cover Image

Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Xie, Zhifei, Wu, Changqiao
Format:	Preprint
Published:	2024
Subjects:	Audio and Speech Processing Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Sound
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
by: Xie, Zhifei, et al.
Published: (2024)

VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
by: Fu, Chaoyou, et al.
Published: (2025)

Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
by: Zhang, Shaolei, et al.
Published: (2025)

Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models
by: Cappellazzo, Umberto, et al.
Published: (2025)

ICASSP 2024 Speech Signal Improvement Challenge
by: Ristea, Nicolae Catalin, et al.
Published: (2024)

Shushing! Let's Imagine an Authentic Speech from the Silent Video
by: Ye, Jiaxin, et al.
Published: (2025)

From Vision to Sound: Advancing Audio Anomaly Detection with Vision-Based Algorithms
by: Barusco, Manuel, et al.
Published: (2025)

From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech
by: Kim, Ji-Hoon, et al.
Published: (2025)

Automated Classification of Phonetic Segments in Child Speech Using Raw Ultrasound Imaging
by: Ani, Saja Al, et al.
Published: (2024)

WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition
by: Li, Feng, et al.
Published: (2024)

CleanUMamba: A Compact Mamba Network for Speech Denoising using Channel Pruning
by: Groot, Sjoerd, et al.
Published: (2024)

See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinement
by: Wang, Jinting, et al.
Published: (2025)

Seeing Your Speech Style: A Novel Zero-Shot Identity-Disentanglement Face-based Voice Conversion
by: Rong, Yan, et al.
Published: (2024)

VAPO: End-to-end Slide-Enhanced Speech Recognition with Omni-modal Large Language Models
by: Hu, Rui, et al.
Published: (2025)

UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction
by: Li, Yadong, et al.
Published: (2026)

Towards Unconstrained Audio Splicing Detection and Localization with Neural Networks
by: Moussa, Denise, et al.
Published: (2022)

Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision
by: Liu, Che, et al.
Published: (2025)

GaussianSpeech: Audio-Driven Gaussian Avatars
by: Aneja, Shivangi, et al.
Published: (2024)

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer
by: Burchi, Maxime, et al.
Published: (2024)

Vision-to-Music Generation: A Survey
by: Wang, Zhaokai, et al.
Published: (2025)

Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content
by: Wu, Sheng, et al.
Published: (2024)

SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model
by: Qian, Xinyuan, et al.
Published: (2024)

Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization
by: Jia, Yanhao, et al.
Published: (2025)

A Small-footprint Acoustic Echo Cancellation Solution for Mobile Full-Duplex Speech Interactions
by: Jiang, Yiheng, et al.
Published: (2025)

DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation
by: Chen, Junming, et al.
Published: (2024)

JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization
by: Liu, Kai, et al.
Published: (2025)

Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities
by: Wang, Siyin, et al.
Published: (2024)

CineSRD: Leveraging Visual, Acoustic, and Linguistic Cues for Open-World Visual Media Speaker Diarization
by: Huang, Liangbin, et al.
Published: (2026)

SyncVoice: Towards Video Dubbing with Vision-Augmented Pretrained TTS Model
by: Wang, Kaidi, et al.
Published: (2025)

Audio-Enhanced Vision-Language Modeling with Latent Space Broadening for High Quality Data Expansion
by: Sun, Yu, et al.
Published: (2025)

OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup
by: Cheng, Xize, et al.
Published: (2024)

VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learning
by: Cheng, Xin, et al.
Published: (2025)

AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation
by: Wang, Le, et al.
Published: (2025)

OmniAudio: Generating Spatial Audio from 360-Degree Video
by: Liu, Huadai, et al.
Published: (2025)

Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder
by: Lau, Kin Wai, et al.
Published: (2026)

Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-Reading
by: Luo, Songtao, et al.
Published: (2023)

LD-LAudio-V1: Video-to-Long-Form-Audio Generation Extension with Dual Lightweight Adapters
by: Zhang, Haomin, et al.
Published: (2025)

Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity
by: Pascual, Santiago, et al.
Published: (2024)

SAVE: Segment Audio-Visual Easy way using Segment Anything Model
by: Nguyen, Khanh-Binh, et al.
Published: (2024)

Mutual Learning for Acoustic Matching and Dereverberation via Visual Scene-driven Diffusion
by: Ma, Jian, et al.
Published: (2024)