Imagen de Portada

Noise-Robust AV-ASR Using Visual Features Both in the Whisper Encoder and Decoder

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Li, Zhengyang, Graave, Thomas, Möller, Björn, Wu, Zehang, Franz, Matthias, Fingscheidt, Tim
Formato:	Preprint
Publicado:	2026
Materias:	Audio and Speech Processing Computation and Language Computer Vision and Pattern Recognition Sound
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Extending Whisper with prompt tuning to target-speaker ASR
por: Ma, Hao, et al.
Publicado: (2023)

mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition
por: Rouditchenko, Andrew, et al.
Publicado: (2025)

Revisiting Acoustic Features for Robust ASR
por: Shah, Muhammad A., et al.
Publicado: (2024)

Diagnostic-Driven Layer-Wise Compensation for Post-Training Quantization of Encoder-Decoder ASR Models
por: Wang, Xinyu, et al.
Publicado: (2026)

Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation
por: Rouditchenko, Andrew, et al.
Publicado: (2024)

Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whisper
por: Thorbecke, Iuliia, et al.
Publicado: (2024)

Quantizing Whisper-small: How design choices affect ASR performance
por: Söhler, Arthur, et al.
Publicado: (2025)

WhisperKit: On-device Real-time ASR with Billion-Scale Transformers
por: Orhon, Atila, et al.
Publicado: (2025)

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding
por: Boccato, Tommaso, et al.
Publicado: (2026)

Overcoming Data Scarcity in Multi-Dialectal Arabic ASR via Whisper Fine-Tuning
por: Özyilmaz, Ömer Tarik, et al.
Publicado: (2025)

Target Speaker ASR with Whisper
por: Polok, Alexander, et al.
Publicado: (2024)

Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder
por: Shakeel, Muhammad, et al.
Publicado: (2025)

Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper
por: Xu, Tianyi, et al.
Publicado: (2024)

A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognition
por: Shi, Xiaohan, et al.
Publicado: (2023)

Transcript-Prompted Whisper with Dictionary-Enhanced Decoding for Japanese Speech Annotation
por: Hu, Rui, et al.
Publicado: (2025)

Whisper Turns Stronger: Augmenting Wav2Vec 2.0 for Superior ASR in Low-Resource Languages
por: Anidjar, Or Haim, et al.
Publicado: (2024)

SQ-Whisper: Speaker-Querying based Whisper Model for Target-Speaker ASR
por: Guo, Pengcheng, et al.
Publicado: (2024)

Are Transformers in Pre-trained LM A Good ASR Encoder? An Empirical Study
por: An, Keyu, et al.
Publicado: (2024)

Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialect
por: Mdhaffar, Salima, et al.
Publicado: (2024)

Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding
por: Zhao, Jiahui, et al.
Publicado: (2024)

WhiSQA: Non-Intrusive Speech Quality Prediction Using Whisper Encoder Features
por: Close, George, et al.
Publicado: (2025)

UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models
por: Fan, Ruchao, et al.
Publicado: (2024)

Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
por: Xie, Yuan, et al.
Publicado: (2026)

Continual Learning Optimizations for Auto-regressive Decoder of Multilingual ASR systems
por: Kwok, Chin Yuen, et al.
Publicado: (2024)

Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understanding
por: Jung, Yeonjoon, et al.
Publicado: (2024)

Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detection
por: Wang, Haoyu, et al.
Publicado: (2024)

AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancement
por: Chou, Ju-Chieh, et al.
Publicado: (2023)

NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
por: Xie, Yuan, et al.
Publicado: (2026)

Visual Cues Support Robust Turn-taking Prediction in Noise
por: Russell, Sam O'Connor, et al.
Publicado: (2025)

Loss Masking Is Not Needed in Decoder-only Transformer for Discrete-token-based ASR
por: Chen, Qian, et al.
Publicado: (2023)

Model-free Speculative Decoding for Transformer-based ASR with Token Map Drafting
por: Ho, Tuan Vu, et al.
Publicado: (2025)

Enhanced Hybrid Transducer and Attention Encoder Decoder with Text Data
por: Tang, Yun, et al.
Publicado: (2025)

LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT
por: Zhuo, Le, et al.
Publicado: (2023)

Exploring SSL Discrete Speech Features for Zipformer-based Contextual ASR
por: Cui, Mingyu, et al.
Publicado: (2024)

MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder
por: Le-Duc, Khai, et al.
Publicado: (2024)

PromptASR for contextualized ASR with controllable style
por: Yang, Xiaoyu, et al.
Publicado: (2023)

A Comparative Study of LLM-based ASR and Whisper in Low Resource and Code Switching Scenario
por: Song, Zheshu, et al.
Publicado: (2024)

Whisper-PMFA: Partial Multi-Scale Feature Aggregation for Speaker Verification using Whisper Models
por: Zhao, Yiyang, et al.
Publicado: (2024)

Spontaneous Speech-Based Suicide Risk Detection Using Whisper and Large Language Models
por: Cui, Ziyun, et al.
Publicado: (2024)

Neural Kalman Filters for Acoustic Echo Cancellation
por: Seidel, Ernst, et al.
Publicado: (2025)